We are family: designing and developing a mobile health application for the San Francisco bay area House Ball and Gay Family communities

Introduction

The House Ball community (HBC) is a community that consists of predominantly African American and Latinx sexual, gender, and ethnic minorities (SGEM) of low socioeconomic status. Since its founding in Harlem in the 1920s, the HBC has expanded rapidly to major US and international cities. The community consists of ‘houses’, or family-like networks, led by mothers and fathers, and the elaborate balls they host. Balls serve as community events where members compete in performative categories and celebrate their SGEM identities. Many HBC-involved youth also form and rely on members of their Gay Families (GFs), led by gay- or transgender-identified individuals who play the role of mothers and fathers. House and GF parents provide guidance and support to their children in a variety of domains. One of the few communities that is welcoming to SGEM-identified individuals, the HBC exists in major cities across the US, and its balls regularly attract large segments of these populations. The social meaning of family has been central to House Ball and Gay Family communities since members have not been historically privileged with supportive biological kin or other support networks in their communities of origin (1). Both communities fulfill a need for young people, many of whom consult with their GF or house parents and siblings on issues related to health that they may not typically discuss with their biological kin, including HIV (2).

Although the HBC has lost members to the HIV epidemic for decades, researchers first documented an HIV prevalence of 17% within the New York HBC in 2004, with 73% being unaware of their serostatus (3). Since then, researchers have documented HIV prevalence of 6% among HBC members in Los Angeles (4), while 27% of those surveyed in the San Francisco Bay Area self-reported living with HIV, reflecting the demographics of the local HIV epidemic which more heavily impacts Black and Latino MSM (5). HIV-related risk behaviors, including condomless anal intercourse, multiple sexual partners, substance use, exchange sex, and low rates of PrEP uptake have been observed in both the HBC and affiliated GF communities across multiple cities (6-9). Social factors such as HIV-related stigma and medical mistrust present challenges to reducing sexual risk behaviors among those who are HIV-negative and engagement in care and treatment for those who are living with HIV (10). Yet, HBC and GF networks also provide HIV-related social support that correlates with HIV testing, condom use, and being aware of HIV medication like pre-exposure prophylaxis (PrEP) (5,11,12), resources that could be harnessed in HIV prevention interventions if they are thoughtfully designed.

Based on 2019 data from Pew Research Center, over 81% of people living in the US have a smartphone, and this ownership is 96% for 18–29-year-old and 92% for 30–49-year-old (13). Mobile health (mHealth) technologies have the capacity to encourage regular HIV testing by helping individuals manage their health needs, facilitate better informed decisions about their health and wellness, develop and enhance their emotional and social support networks, and develop positive health behaviors (14). mHealth technologies can also connect healthcare providers and researchers to populations that may be hard-to-reach or less-engaged in care, improve the quality of care, and decrease healthcare costs. More specifically, clinical studies have shown that receiving mHealth interventions is associated with changing indicators of disease management, including increasing social support, self-care behaviors, symptom management, self-efficacy, and disease-related knowledge (15,16). Several systematic reviews reveal the availability of mHealth interventions and applications on the app store; however, the authors note the infrequency of downloads and low quality due to lack of desirable features (17,18). Additionally, studies show that despite a large number of available health apps, only a fraction are downloaded and used (19). Average retention rate for mobile apps are approximately 30% after 90 days, and about 21% of users abandon an application after a single use (20). Therefore, these studies highlight the need for mHealth research that includes the end user at every step of product discovery, from conceptualization to implementation, in a process of co-creation (21). This process has been recognized as a responsible means to establish effective mHealth technologies (21,22). The co-creation process can provide valuable insight into how the end users engage and interact with technologies which can lead to timely and relevant solutions to a clinical problem, and ultimately may result in improved clinical outcomes. Due to constant modifications in technology and sociocultural developments of the end user, co-creation is not a static process and requires ongoing updates, fixes, and improvements.

Given the ubiquity of smartphone usage and the potential of mHealth apps in changing health behaviors, we believe that a culturally-tailored and theory-driven intervention that is delivered via mobile technology has the potential to increase testing and linkage to care among the House Ball and Gay Family communities. Therefore, the purpose of our study was to examine the discovery of an mHealth solution to encourage the House Ball and Gay Family communities to seek regular testing if HIV negative and link to care if positive using a co-creation process. This mHealth solution was a component of a larger intervention called We Are Family which is grounded in the Information, Motivation, Behavioral Skills Model (IMB) (23-25) and includes group sessions and activities at balls.

Methods

Data collection took place between August 2016 and December 2017. We conducted 45 in-depth interviews with HBC leaders, HBC members, and GF members who attended balls. A team consisting of a white female anthropologist, and two gay men of color research assistants conducted the interviews. Eligible participants were: part of a House or a Gay Family, gay, bisexual, transgender or otherwise non-gender conforming identified, sexually active as measured by anal intercourse with a man in the past year, San Francisco Bay Area residents, and at least 18 years old. Participants were recruited using snowball sampling and flyers were posted at LGBT community events as well as HBC events. A research assistant screened interested individuals and invited those eligible to participate in either the interviews or the workshops depending on the timing of recruitment. Interviews lasted 60–120 minutes and were recorded and transcribed verbatim. Topics included HIV-related health needs, suggestions for mhealth components, current usage of apps and desirable features, and wishes around privacy and security. Questions included: (I) Can you tell me about a time you used an online or an electronic resource to find information or support on HIV prevention, including testing and treatment? (II) What kind of information or functionality do you wish you had in an app? (III) What electronic devices are available to you in your day to day life?, and (IV) Would you prefer a web-based site that has this content and/or tablet and mobile-phone apps?

Interviews took place in private offices at the University of California, San Francisco (UCSF), the California Prevention Education Project (CAL-PEP), a community-based organization in Oakland, CA that collaborated on the project, or at a private location of the participant’s choosing. Interviews proceeded until theoretical saturation was achieved, with no new themes emerging in our analysis (26). Study participants all provided verbal consent and received US$40 to compensate for time and travel. All study procedures were reviewed and approved by the UCSF Institutional Review Board.

To explore our data, we followed procedures from thematic analysis, developing and defining thematic codes to capture data segments related to HIV-related health needs and resources, desirable app features, privacy/security concerns and preferred platforms/devices (27). Themes were developed using both a deductive (based on topics in the guides) and an inductive (emergent findings) approach to characterize the data. Meeting regularly, the authors organized the thematic areas into a defined set of codes. Interview transcripts were then uploaded to an online qualitative data analysis program, Dedoose (Version 8.0.35, 2018). We then applied the codes to a subset of transcripts, until code application was consistent across coders. The research assistant then coded the remaining transcripts, and he and the senior author conducted cross-case comparisons across the entire data set to ensure consistency of the coding structure. Analytic memos summarized meeting discussions and coding decisions. In this paper, participants are identified using unique identification numbers.

Following the same recruitment, enrollment, and consent processes outlined above, we also convened participants for two workshops (each meeting twice) with a total of 15 participants who were divided into two groups for design workshops. In keeping with the IMB model, the purpose of the workshops was to co-design a digital tool to help provide information, reduce stigma, and locate resources for We Are Family participants. Each group attended two workshops for a total of four workshops.

Co-design or Participatory Design is a sub-set of practices from the broader field of human-centered design (28,29) particularly suited to engaging marginalized communities in healthcare information technology development in community settings (30). With a team drawn from diverse backgrounds (anthropology, clinical and implementation science research, community health, and digital product design) our iterative design process followed an approach that involved four phases (31): (I) Discover: review of the transcribed interviews, exploration of the questions and themes with the Community Advisory Board; (II) Define: understanding the current digital landscape, multiple iterations of the feature set and product definition as well as co-design activities that enabled participants to generate content (e.g., images, videos and text) for the product; (III) Design: translation of interviews and workshops findings into a digital product, detailed review of choices such as colors, images, and micro-copy (button labels, section titles, etc.), creating mock-ups, as well as hand-off for build, and (IV) Deliver: creation of a mobile optimized Drupal Community site.

In the first workshop, we described the overall intervention and the goals of the digital tool. Additionally, we presented existing digital tools and campaigns as a starting point for discussion and asked the group to share what resonated with them and why. They looked at campaigns focusing on sexual health, HIV, and the LGBT community, websites that crowd-source and collect feedback, online games, social media, and educational videos. We engaged in discussions exploring communication style and tone, content, gender and race representation, privacy, and functionality. The goal of this exercise was for participants to provide direction for the project, ensure that the product brought up desired emotions and behaviors, and identity potential features to adopt or avoid.

Participants also took part in a co-creation sketching exercise. We asked them to generate ideas for games and quizzes that could be evolved and integrated into the digital product. By sharing their rational for specific features, participants identified goals for the digital product. These ideas, in conjunction with notes taken during the workshops allowed the design team to develop several different mock-ups of the app to be presented and shared with participants as part of the second workshop.

In the second workshop, we collectively prioritized and converged on a design solution. Using Visual Preference Testing (32), participants voted on their preferred overall look and feel from three design directions presented as prototypes on mobile phones. Participants were also prompted for feedback on specific features, content elements and functionality by collecting their ideas and reactions on post-its, which were stuck onto design mock-ups. We facilitated a group discussion to understand which ideas were most popular and why, as well as relevant to the study aim. This exercise informed final decisions about what to incorporate into the digital tool’s design. The chosen design direction was revised incorporating feedback from this second workshop. A clickable prototype was created to validate functionality by the study team before developing the mobile optimized Drupal Community site.

Results

We conducted one-time individual qualitative interviews with 45 individuals and four workshops with 15 individuals (each invited to participate in two workshops to co-design the app for a total of four workshops). Participants were primarily male, African American, gay-identified, and had taken some college classes (Table 1).

Table 1 Characteristics of study participants in individual interviews and focus groups
Full table

Participants noted that they used YouTube, Google Maps, Facebook/Instagram, Snapchat, and Sound Cloud apps. The main reasons for using these apps were interest in watching competitions, finding locations of Balls, promoting their own businesses (e.g., dance classes), getting followers, and exposing themselves to the music industry. Participants noted that they particularly enjoyed the opportunity to be creative with art on various apps such as Snapchat and Instagram.

Participants discussed the creation of app content that would partner with the HBC and support their creative talents. A major focus of discussion brought up by participants was the need for the app to represent the HBC. Some suggestions for this representation were in the form of app users sharing testimonies and experiences that could help others going through a similar situation, sharing mutual history of pride and the HBC, inspiring others with personal stories and decreasing social isolation, and sharing the legacies of House Ball icons and legends who lived with HIV. Despite the request from most participants to have a space to share their stories, several did not feel as strongly and felt that their lived experiences and identities would be oversimplified by only sharing their HIV-related stories. Table 2 includes exemplary quotes from participants related to these themes.

Table 2 Exemplary quotes from individual qualitative interviews by theme and sub-theme
Full table

When asked about ways of increasing engagement in app use, participants gave several suggestions. Some of these suggestions included posting upcoming events and balls in the app, using the app as a tool to help people meet while receiving health messages, involving house parents in app use, providing a point system or incentives for app use by tapping into the competitive nature of the HBC, and using Ballroom trivia in the app.

Participants suggested several features for the app, including information and education on HIV transmission; links to resources such as HIV testing, lube and condoms, PrEP, and other health-related services; and even locating a counselor or doctor. Other ideas included a virtual bulletin board where users could post questions on HIV or other topics anonymously. Finally, participants favored the idea of being able to rate the clinics and community organizations so that other app users could identify places to go for various health needs.

Participants also had several additional requirements for an app, for example that it use minimal space on their phones, that it has a disclaimer about potential inadvertent disclosures, and that it be anonymous. One participant noted: “That’s the problem with the closet is that it’s so easy to out yourself and subject yourself to the overall [form of online] violence” (PT037).

In general, the main downsides of technology were noted to be the fact that it may be too exposing if the user overshares, the virtual realm can become a harsh and racist space, it can lead to negativity, and it can lead to superficiality of interactions as noted by one participant: “Mandating superficiality of interaction. What that turns into is a meat market.. I have to compress the complexity of me into a 2D image and [social media name] creates this whole high intensity, high stakes politics of desirability that affects people’s psyches really easily, and with really quick brush strokes” (PT014). Participants also noted specific challenges with using an app, such as their worry about it becoming a hookup app, the negativity and trolling that may occur in an app, and that they already have too many apps and do not need more and that another app may be redundant.

Table 1 details the characteristics of the 15 individuals who participated in the two workshops that were repeated twice. The following are themes that emerged from the workshops and informed the final design:

• Mobile and Modern: Participants expressed an interest for a digital piece that was bold, bright, modern, and modular. They noted that mobile access was critical, in that it had to be easy to use with minimal text; however, they wanted the ability to dig deeper for more information. A Frequently Asked Questions section was requested to provide short, conversational educational content that was written in lay language.
• Relatable, Raw, and “Reflective of Me”: Participants were interested in seeing relatable faces from the community, including a variety of races, body types, and gender identities, who had some celebrity status. They wanted to hear people telling their personal unscripted stories. “Show me real people - not like on a set,” one participant said. Participants desired videos that addressed how to have difficult conversations, such as how to disclose a positive HIV status. In addition, they wanted to hear from uplifting people and about the power of helping each other in the community. “It’s empowering to empower other people.”
• Positive and Fun: Participants favored a positive, but serious tone. They wanted imagery that was provocative, playful, and sex-positive because “sexy is appealing.” Funny and playful elements such as memes, pop-culture, and fashion references were suggested to help engagement with the use of the tool.
• Community-generated Content: Participants wanted to see content generated by the community on an ongoing basis to keep it fresh and relevant. When asked, participants said they were willing to submit content for videos, quotes, and Instagram feeds, as well as take polls and quizzes. Timely results to games and quizzes could help with user retention. Participants shared services they wanted included in the resource finder (e.g., Yelp), desired tool functionality (e.g., Maps), and specific aspects of services important to them such as, provider bedside manners, trans-friendly environment and the use of oral swabs for HIV testing. They also wanted the ability to rate and review specific aspects of the services.
• Safe Space: Participants wanted to ensure the tool was a positive, trusted, and digital space to share and find information, and that it was free of hook-ups or “nasty” comments. Therefore, allowing a moderator to approve posts, vet reviews of services, and preserve user anonymity were considered to be critical. To avoid duplication, it was decided that the We Are Family Facebook page was a better place to post events and allow for interaction among participants. Lastly, we heard that the digital tool would be most effective if not branded as a campaign, but rather as a long-term, evolving space for the community.

During the co-creation process, participants voiced concerns about the cost of developing the platform, security and privacy, and the overall sustainability of the app. Therefore, the design and development team chose an open source platform (i.e., Drupal) specifically tailored to community driven sites. The actual website (https://wearefamily.ucsf.edu/) is only available to users with login information and requires Secure Sockets Layer or SSL, which is used to send information securely over the Internet. This website has gone through an institutional security and risk assessment. Additionally, we minimized data collected to create an initial profile to comply with institutional standards about data collection (https://it.ucsf.edu/knowyourdata).

Based on the individual interviews and workshops, we considered three features to the We Are Family app: (I) Find Services, (II) The Facts, and (III) Real Stories (Figure 1). Find Services was based on the users’ interest in rating services and the interest in app such as Yelp (Figure 2). This feature includes services related to health, testing, mental health, substance use, food, housing, jobs, and clothing in the San Francisco Bay Area. Users can write reviews and rate the service using a five-star rating system. The Facts includes weekly True/False or multiple choice questions related to HIV prevention and treatment (Figure 3). The responses to these questions are revealed in weekly polls that show how users voted and provide correct information. Finally, Real Stories stems from the desire of the users to share their stories and the interest of these communities in videos and performances (Figure 4). These features map on to the IMB model by providing needed information and motivation to result in a behavior change.

Figure 1 WAF home page.

Figure 2 Types of services provided in WAF app.

Figure 3 Weekly quiz in WAF app.

Figure 4 Videos and stories listed in WAF app.

Discussion

Our interviews and co-creation process resulted in an app that revealed the wishes and needs of members of the House Ball and Gay Family communities. Participants wanted an mHealth app that reflected the communities that they are a part of, a safe space to share stories, and demonstrated individuals who looked and spoke like them navigating difficult situations and conversations, such as disclosing a positive HIV serostatus. Given the high levels of medical mistrust and encounters of homophobia and transphobia in medical and other health-related service settings, our participants also wanted to be able to rate and review community resources to ensure that members of their chosen families would receive high quality, respectful care. Links to information, including fun games that would quiz users on their knowledge around HIV transmission and care were also suggested by participants with correct answers being placed in a repository of “Q and A” for participants to look up later. Finally, making sure that the app was private, that it allowed users to be anonymous, and that it did not take up a great deal of space on phones were all deemed to be necessary aspects of the app.

Several studies have examined and tested mHealth app use among people living with HIV; however, only a few have examined the desires of the users and involved them to inform the design of the mHealth tool in a co-creation process (33-37). Despite the availability of numerous mHealth apps in the app store (17,38), most are infrequently downloaded and are considered low quality by users due to lack of desirable features (39), and very few are actually used (19). The low frequency of downloads and low ratings have also been reported in a review of apps for care and prevention of HIV and other sexually transmitted infections (40). These studies demonstrate the need for research that involves the end users at every step of the design, development, and implementation processes to increase engagement and continued use of the app. One review of mHealth interventions to support self-management among people living with HIV noted that only 50% of the reviewed interventions had any involvement from the target populations, which spanned from the use of workshops to iterative testing and development (18). Additionally, none of the reviewed mHealth apps for people living with HIV included all identified key desired functions.

The development of effective interventions to address HIV testing, linkage, and engagement in care among SGEM and other communities with high rates of HIV-related stigma is a research priority. While interventions have been developed in partnership with the HBC especially, addressing PrEP uptake (11) as well as sexual risk behavior (41,42), these efforts rely on in person delivery modalities and have not explored the use of technology to reach this audience. Due to the ubiquity of mobile phones (13), mHealth interventions can offer direct, expeditious, and cost-effective modes of intervention delivery. Other studies are currently under way for the development of an HIV prevention app for men who have sex with men (43), HIV viral suppression among young men who have sex with men and transgender women (36,44), and engagement in care among youth and young adults living with HIV (45,46).

The significance and innovation of our research is the methodical formative research with individuals from the House Ball and Gay Family communities to co-design an mHealth app for HIV testing and linkage to care. Additionally, we were mindful of including participants from across the San Francisco Bay Area and along the gender continuum, racial/ethnic groups, and House and GF parents and children. However, our research has several limitations that should be noted. We only had one person who was newly HIV diagnosed; therefore, we did not have sufficient representation from the entire HIV care continuum. Additionally, we focused on the San Francisco Bay Area House Ball and Gay Family communities; therefore, our results may not be generalizable to other populations or House Ball and Gay Family communities in other locations. However, given the lack of data on these hard-to-reach populations, we believe our data adds to the knowledge base with regard to the requirements and features of mHealth apps for the House Ball and Gay Family communities.

As part of this research, we will continue to collaborate with members of the San Francisco Bay Area House Ball and Gay Family communities in a co-creation process to iteratively develop and refine an mHealth app that will encourage these communities to seek regular testing if HIV negative and link to care if positive. In future trials, by collaborating closely with community members, we will examine the feasibility, acceptability, and usability of the use of an mHealth app along with in person educational and community-building activities to impact HIV-related health outcomes for these communities. If this proves promising, we will seek to conduct an efficacy trial to examine the impact such an intervention on HIV testing (for those who are HIV seronegative) and linkage to care (for those living with HIV). Challenges to be examined in future research include participant engagement with technology use, mobile phone turnover (due to frequent changes phones or phone numbers, loss or theft of phone, disconnection of phone service, etc.), data usage and WIFI connectivity, need for approval of posts by moderator, and privacy and security of the app.

Conclusions

In a community with tight social and sexual networks, HIV-related stigma that prevents regular testing and care engagement, and therefore high rates of undiagnosed and untreated HIV, it is essential to focus on identifying and treating new and out-of-care cases. By understanding the needs of the end user, we are able to develop a product that has a higher chance of impact, engagement, and sustainability. Using an mHealth app as part of a larger HIV prevention intervention that reflects the social support and relationships within existing House Ball and Gay Family communities allows those youth most at risk for HIV-related health disparities to gain access to HIV testing, and linking and re-engaging young people to care.

Acknowledgments

The authors would like to thank the California HIV/AIDS Research Program (CHRP award number HD15-SF-060), the study participants, the members of our Community Advisory Board, CAL-PEP, and the developer Joe Miller from Everything Inspired.

Funding: The California HIV/AIDS Research Program (CHRP award number HD15-SF-060).

Footnote

Provenance and Peer Review: This article was commissioned by the Guest Editors (Carinne Brody and Sarah Sullivan) for the series “Digital Interventions for Hard-to-reach Populations” published in mHealth. The article was sent for external peer review organized by the Guest Editor and the editorial office.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/mHealth-19-234). The series “Digital Interventions for Hard-to-reach Populations” was commissioned by the editorial office without any funding or sponsorship. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Study participants all provided verbal consent and received US$40 to compensate for time and travel. All study procedures were reviewed and approved by the UCSF Institutional Review Board.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Heaphy B. Family, kinship and citizenship: Change and continuity in LGBQ lives. In: Aggleton P, Cover R, Leahy D et al., editors. Youth, Sexuality and Sexual Citizenship. London: Routledge; 2019. p. 19-31.
2. Arnold EA, Bailey MM. Constructing Home and Family: How the Ballroom Community Supports African American GLBTQ Youth in the Face of HIV/AIDS. J Gay Lesbian Soc Serv 2009;21:171-88. [Crossref] [PubMed]
3. Murrill CS, Liu KL, Guilin V, et al. HIV prevalence and associated risk behaviors in New York City's house ball community. Am J Public Health 2008;98:1074-80. [Crossref] [PubMed]
4. Kipke MD, Kubicek K, Supan J, et al. Laying the groundwork for an HIV prevention intervention: a descriptive profile of the Los Angeles House and Ball communities. AIDS Behav 2013;17:1068-81. [Crossref] [PubMed]
5. Arnold EA, Sterrett-Hong E, Jonas A, et al. Social networks and social support among ball-attending African American men who have sex with men and transgender women are associated with HIV-related outcomes. Glob Public Health 2018;13:144-58. [Crossref] [PubMed]
6. Phillips G 2nd, Peterson J, Binson D, et al. House/ball culture and adolescent African-American transgender persons and men who have sex with men: a synthesis of the literature. AIDS Care 2011;23:515-20. [Crossref] [PubMed]
7. Castillo M, Palmer BJ, Rudy BJ, et al. Creating partnerships for HIV prevention among YMSM: the Connect Protect(R) Project and House and Ball Community in Philadelphia. J Prev Interv Community 2012;40:165-75. [Crossref] [PubMed]
8. Schrager SM, Latkin CA, Weiss G, et al. High-risk sexual activity in the House and Ball community: influence of social networks. Am J Public Health 2014;104:326-31. [Crossref] [PubMed]
9. Lemos D, Hosek SG, Bell M. Reconciling Reality with Fantasy: Exploration of the Sociocultural Factors influencing HIV Transmission among Black Young Men who have Sex with Men (BYMSM) within the House Ball Community: A Chicago Study. J Gay Lesbian Soc Serv 2015;27:64-85. [Crossref] [PubMed]
10. Galindo GR. A loss of moral experience: understanding HIV-related stigma in the New York City House and Ball Community. Am J Public Health 2013;103:293-9. [Crossref] [PubMed]
11. Young LE, Jonas AB, Michaels S, et al. Social-structural properties and HIV prevention among young men who have sex with men in the ballroom house and independent gay family communities. Soc Sci Med 2017;174:26-34. [Crossref] [PubMed]
12. Levitt HM, Horne SG, Freeman-Coppadge D, et al. HIV Prevention in Gay Family and House Networks: Fostering Self-Determination and Sexual Safety. AIDS Behav 2017;21:2973-86. [Crossref] [PubMed]
13. Pew Research Center. Mobile Fact Sheet. 2019. Available online: https://www.pewinternet.org/fact-sheet/mobile/. Accessed 10/8/2019.
14. World Health Organization. mHealth New horizons for health through mobile technologies. In: Global Observatory for eHealth series. 2011. Available online: https://www.who.int/goe/publications/goe_mhealth_web.pdf. Accessed 10/20/2019.
15. Kato PM, Cole SW, Bradlyn AS, et al. A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial. Pediatrics 2008;122:e305-17. [Crossref] [PubMed]
16. Muessig KE, Pike EC, Fowler B, et al. Putting Prevention in Their Pockets: Developing Mobile Phone-Based HIV Interventions for Black Men Who Have Sex with Men. AIDS Patient Care STDS 2013;27:211-22. [Crossref] [PubMed]
17. Santo K, Richtering SS, Chalmers J, et al. Mobile Phone Apps to Improve Medication Adherence: A Systematic Stepwise Process to Identify High-Quality Apps. JMIR Mhealth Uhealth 2016;4:e132. [Crossref] [PubMed]
18. Cooper V, Clatworthy J, Whetham J, et al. mHealth Interventions To Support Self-Management In HIV: A Systematic Review. Open AIDS J 2017;11:119-32. [Crossref] [PubMed]
19. Aitken M, Lyle J. Patient adoption of mHealth: use, evidence and remaining barriers to mainstream acceptance. IMS Institute for Healthcare Informatics, Parsippany, NJ. 2015. Available online: http://microsites.schott.com/d/studentchallenge/346d3edb-327f-42ad-b3b5-d859b8c4cf6f/1.0/ims_report_patient_adoption_of_mhealth.pdf. Accessed 10/20/2019.
20. Perro J. Mobile Apps: What’s A Good Retention Rate? Localytics Blog. 2018. Available online: http://info.localytics.com/blog/mobile-apps-whats-a-good-retention-rate. Accessed 3/15/2019.
21. Bate SP, Robert G. Bringing user experiences to health care improvement: The concepts, methods and practices of experience-based design. Oxford: Radcliffe Publishing: 2007.
22. Wherton J, Sugarhood P, Procter R, et al. Co-production in practice: how people with assisted living needs can help design and evolve technologies and services. Implementation Science 2015;10:75. [PubMed]
23. Fisher JD, Fisher WA, Amico KR, et al. An information-motivation-behavioral skills model of adherence to antiretroviral therapy. Health Psychol 2006;25:462-73. [Crossref] [PubMed]
24. Fisher JD, Fisher WA, Bryan AD, et al. Information-motivation-behavioral skills model-based HIV risk behavior change intervention for inner-city high school youth. Health Psychol 2002;21:177-86. [Crossref] [PubMed]
25. Fisher JD, Fisher WA, Shuper PA. The Information-Motivation-Behavioral Skills Model of HIV Preventive Behavior. In: DiClemente R, Crosby R, Kegler M, editors. Emerging Theories in Health Promotion Practice and Research. San Francisco, CA: Jossey Bass; 2000. p. 22-63.
26. Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Methods 2006;18:59-82. [Crossref]
27. Miles MB, Huberman AM. Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage Publications; 1994.
28. Kushniruk A, Nohr C. Participatory Design, User Involvement and Health IT Evaluation. Stud Health Technol Inform 2016;222:139-51. [PubMed]
29. Clemensen J, Larsen SB, Kyng M, et al. Participatory design in health sciences: Using cooperative experimental methods in developing health services and computer technology. Qual Health Res 2007;17:122-30. [Crossref] [PubMed]
30. Chen E, Leos C, Kowitt SD, et al. Enhancing Community-Based Participatory Research Through Human-Centered Design Strategies. Health Promot Pract 2020;21:37-48. [PubMed]
31. Design Council. What is the framework for innovation? Design Council's evolved Double Diamond. 2019. Available online: https://www.designcouncil.org.uk/news-opinion/design-process-what-double-diamond. Accessed 10/20/2019.
32. Visual preference testing. 18F Methods. Available online: https://methods.18f.gov/validate/visual-preference-testing/. Accessed 11/19/2019.
33. Schnall R, Bakken S, Rojas M, et al. mHealth Technology as a Persuasive Tool for Treatment, Care and Management of Persons Living with HIV. AIDS Behav 2015;19:81-9. [Crossref] [PubMed]
34. Horvath KJ, Alemu D, Danh T, et al. Creating Effective Mobile Phone Apps to Optimize Antiretroviral Therapy Adherence: Perspectives From Stimulant-Using HIV-Positive Men Who Have Sex With Men. JMIR Mhealth Uhealth 2016;4:e48. [Crossref] [PubMed]
35. Ramanathan N, Swendeman D, Comulada WS, et al. Identifying preferences for mobile health applications for self-monitoring and self-management: Focus group findings from HIV-positive persons and young mothers. Int J Med Inform 2013;82:e38-46. [Crossref] [PubMed]
36. LeGrand S, Muessig KE, McNulty T, et al. Epic Allies: Development of a Gaming App to Improve Antiretroviral Therapy Adherence Among Young HIV-Positive Men Who Have Sex With Men. JMIR Serious Games 2016;4:e6. [Crossref] [PubMed]
37. Saberi P, Siedle-Khan R, Sheon N, et al. The Use of Mobile Health Applications Among Youth and Young Adults Living with HIV: Focus Group Findings. AIDS Patient Care STDS 2016;30:254-60. [Crossref] [PubMed]
38. Muessig KE, Nekkanti M, Bauermeister J, et al. A systematic review of recent smartphone, Internet and Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep 2015;12:173-90. [Crossref] [PubMed]
39. Marent B, Henwood F, Darking M, et al. Development of an mHealth platform for HIV Care: Gathering User Perspectives Through Co-Design Workshops and Interviews. JMIR Mhealth Uhealth 2018;6:e184. [Crossref] [PubMed]
40. Muessig KE, Pike EC, LeGrand S, et al. Mobile Phone Applications for the Care and Prevention of HIV and Other Sexually Transmitted Diseases: A Review. J Med Internet Res 2013;15:e1. [Crossref] [PubMed]
41. Castillo M, Hosek SG. Project Posse: Health Promotion, Recruitment and Engagement of the House/Ball and Kiki Communities for Hiv Prevention. J Adolesc Health 2018;62:S20. [Crossref]
42. Hosek S, Castillo M, Hotton A, et al. Comparison of two distinct house ball communities involved in an HIV prevention study: Baselinedata from the POSSE project. J HIV AIDS Soc Serv 2019;18:399-416. [Crossref]
43. Schnall R, Rojas M, Bakken S, et al. A user-centered model for designing consumer mobile health (mHealth) applications (apps). J Biomed Inform 2016;60:243-51. [Crossref] [PubMed]
44. Hightow-Weidman L, Muessig K, Knudtson K, et al. A Gamified Smartphone App to Support Engagement in Care and Medication Adherence for HIV-Positive Young Men Who Have Sex With Men (AllyQuest): Development and Pilot Study. JMIR Public Health Surveill 2018;4:e34. [Crossref] [PubMed]
45. Erguera XA, Johnson MO, Neilands TB, et al. WYZ: a pilot study protocol for designing and developing a mobile health application for engagement in HIV care and medication adherence in youth and young adults living with HIV. BMJ Open 2019;9:e030473. [Crossref] [PubMed]
46. Whiteley L, Brown L, Lally M, et al. A Mobile Gaming Intervention to Increase Adherence to Antiretroviral Treatment for Youth Living With HIV: Development Guided by the Information, Motivation, and Behavioral Skills Model. JMIR Mhealth Uhealth 2018;6:e96. [Crossref] [PubMed]