Understanding the potential implementation determinants of Our Plan: a couples-based digital human immunodeficiency virus prevention intervention for same-gender male couples

Introduction

In the United States (U.S.), sexual minority men (i.e., gay, bisexual, and other men who have sex with men) continue to be disproportionately burdened by the human immunodeficiency virus (HIV) epidemic (1). Modeling studies estimated that one- to two-thirds of new HIV infections among sexual minority men occurred in the contexts of primary partnerships (2,3). As a result of these high HIV incidence estimates, there have been several couples-based HIV prevention intervention studies designed and conducted with adult, cisgender sexual minority male couples (4-8), as well as with younger sexual minority men where their partners may or may not have participated (i.e., couples-focused) in the intervention (9-11).

Despite the promise of couples-based HIV prevention interventions deemed efficacious, there has been limited uptake of implementing them in person at community-based organizations. Couples HIV testing and counseling (CHTC) has been widely available in the U.S. since 2015, including the provision of and access to training and technical assistance for organizations aiming to provide this service. However, implementation barriers to sustainably provide CHTC have been documented, ranging from a lack of leadership support, limited time to train counselors, and funding and resource constraints, to the inability to integrate CHTC into existing services (12).

Simultaneously, there has been a proliferation in the development and use of digital technologies to improve health [e.g., native applications (apps), webapps, telephone/video conference, social media websites], including for HIV prevention-care continua outcomes, as a mechanism to help circumvent barriers to in-person interventions and services (13). Several couples-based and couples-focused digital health interventions (DHIs) have been developed for sexual minority male couples (6,8-10,14). DHIs have the potential to offer many benefits, such as overcoming geographic and economic access barriers, convenience, and privacy (15,16). DHIs also have the potential to overcome implementation barriers by minimizing the need for certain resources, such as counselors, compared to couples-based interventions provided in person. To date, few, if any, couples-based DHIs designed to address HIV prevention and care continuum outcomes have demonstrated effectiveness in real world contexts or have achieved widespread use (e.g., 2GETHER, ReACH2Gether, Stronger Together), highlighting the urgent need to investigate what and how determinants of implementation are impacting the state of couples-based DHIs.

Our Plan is a DHI that addresses the HIV and sexually transmitted infection (STI) prevention-care needs of adult, cisgender male-male dyads who are in a new relationship, defined as being 12 months or less in duration. Our Plan is for dyads who represent a variety of romantic-sexual partnerships, ranging from boyfriends and couples to more casual ‘situationships’ of ongoing friends with benefits (FWBs) and sex buddies. Prior research has identified that the first year of romantic-sexual partnerships may present partners with heightened vulnerabilities to HIV (17). One study found considerable variation of when sexual minority men and their partners engage in condomless anal sex (CAS), share their HIV serostatus, and form a sexual agreement—often, all happening within the first 3 months and definitively within the first 12 months (18). Sometimes conversations and decisions about HIV serostatus and sexual agreements happened within the same day or after CAS occurred between the couple (18). For these reasons, Our Plan is designed for new romantic-sexual partnerships as a mechanism for partners to concurrently and collaboratively learn—essential skills and topics about their sexual health and relationship needs with the end goal of couples developing a comprehensive HIV/STI prevention-care plan to use and adhere to over time. Our Plan also provides partners with access to community resources and information about local HIV prevention-care services through a geo-locator resource finder. To help ensure the protection of participant data, Our Plan was deployed on Microsoft Azure using secure servers with end-to-end encryption protocols. Access to Our Plan was fortified using multi-factor authentication paired with robust password expiration policies. In addition, the web development staff continuously monitored the DHI to ensure Health Insurance Portability and Accountability Act (HIPAA) compliance and data integrity.

Acquired immunodeficiency syndrome (AIDS) service organizations (ASOs) and community based organizations (CBOs) have been critical in addressing the HIV epidemic, particularly in Ending the HIV Epidemic (EHE) jurisdictions where public resources including the availability of medical care, culturally affirming providers, and staffing and support services are often limited (19). Evidence also demonstrates that counties with the highest HIV incidence and the least optimal geographic access to HIV prevention and care services are in locations deeply entrenched in systemic racism and other forms of oppression (20). These organizations have played an essential role in addressing the HIV epidemic for decades and will continue to be integral to achieving EHE goals given their proximity and connections to communities most devastated by the HIV epidemic (21), and are critical in implementing new strategies for HIV prevention and treatment, and critical provision of addressing structural vulnerabilities (e.g., housing, insurance navigation, transportation) driving HIV inequities (22). Notably, many organizations were able to transition services virtually using telehealth to continue providing care for their clients during the coronavirus disease 2019 (COVID-19) pandemic (23,24), which provides potential support for the feasibility of implementation of DHIs such as Our Plan.

While acceptability, feasibility, and preliminary efficacy of couples-based DHIs for HIV/STI prevention-care have been demonstrated in research contexts (6,9,10,14), less is known about the barriers and facilitators for implementation within community-based organizations. Wide-scale uptake and sustainability of DHIs for HIV/STI prevention among couples in real world contexts requires a better understanding of what specific determinants may help or hinder implementation. The overarching goal of the current study sought to identify implementation determinants of providing Our Plan within the realm of ASOs and CBOs. Qualitative interviews were conducted with organization leaders, healthcare providers, and staff at ASOs and CBOs to identify themes related to the implementation of Our Plan with their patients.

Methods

Participants and procedures

Qualitative interviews were conducted via Zoom with organization leaders, providers, and staff at ASOs and CBOs across the U.S. Participants at the organizations were recruited through convenience sampling methods, whereby they learned about the study from flyers and invitation emails. The research team first identified a range of ASOs and CBOs by using their personal networks. After those options were exhausted, the research team cross-referenced the EHE jurisdictions with ASOs and CBOs that were familiar with the needs of sexual minority men to identify additional organizations to contact for recruitment purposes. Those who were interested in the study would either call or email the research team to learn more about the study and screen for eligibility then or complete the eligibility screener online. Those who were interested and deemed eligible were then scheduled for a one-time qualitative interview conducted over Zoom.

Participants were deemed eligible by meeting all the following inclusion criteria: (I) being at least 18 years of age; (II) currently employed at least part-time for 6 months at a CBO, ASO, or clinic that provides STI treatment and/or HIV prevention-care services; (III) having a position with their current employer as a medical provider, leadership, case manager, HIV test counselor, recruitment/outreach, or other staff (e.g., outreach worker); (IV) comfortable participating in a video-based interview in English or Spanish over Zoom; and (V) able to provide informed consent.

All participants provided informed consent to participate and to have their respective interview audio recorded, transcribed, checked for accuracy, and de-identified for research purposes. Semi-structured interviews were conducted by three of the authors between April and August 2022. The interviews lasted approximately 1 to 2 hours and participants received a $50 Amazon gift card for their time. The study design and associated procedures were approved by Florida International University Institutional Review Board (No. IRB-21-0485). The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013).

Qualitative interviews

The interview items and follow-up questions were guided by the Consolidated Framework for Implementation Research (CFIR), which explore determinants of the implementation of innovations (25,26). The interviews first inquired about participants’ knowledge and experiences with DHIs, followed by a description of Our Plan and sharing two brief videos that lasted approximately 2–3 minutes each. The first video explained the purpose and some key features of Our Plan whereas the second video provided a demonstration of the backend portal of Our Plan, which highlighted how data would be managed and tracked. After the videos, participants were then asked questions about their opinion of Our Plan guided by the CFIR domains. Table 1 provides example questions asked in the interviews that map onto the CFIR domain.

Qualitative analysis

Thematic content analyses (27) were conducted by the first five authors and Microsoft Excel was used to organize the data. First, the team developed a thematic matrix based on the CFIR interview guide. The interviewers coded each transcript and created a series of main and sub-category codes. Participant responses were categorized by each code. The thematic matrix was reviewed for completion by all five coders. Then, each coder independently reviewed participants’ responses by the updated CFIR domains (25) and engaged in a consensus process to resolve discrepancies through team discussions. After reaching consensus, each coder refined their coding. The themes were then reviewed as a team and the final structure was reviewed and confirmed by all authors.

Results

Sample characteristics

The final sample consisted of 40 participants who represented ASOs and CBOs in 13 states serving populations in EHE jurisdictions. Specifically, participants resided in Alabama (n=3), California (n=4), Florida (n=2), Illinois (n=2), Indiana (n=2), Michigan (n=11), Missouri (n=4), Nevada (n=1), North Carolina (n=1), Pennsylvania (n=1), Rhode Island (n=1), Texas (n=2), and Washington, DC (n=6). The majority of participants (45%) were in leadership/management positions, 17.5% providers, 12.2% case managers, and 27.5% HIV/STI test counselors, pre-exposure prophylaxis (PrEP) navigators, or outreach workers. Participants ranged in age from 22 to 66 years [mean (M) =35 years, standard deviation (SD) =10 years] and were racially diverse (i.e., 40% Black, 40% White, 7.5% multiracial, 2.5% Asian) and 17% identified as Hispanic/Latinx. Half of the participants identified as cisgender women, 42.5% as cisgender men, and 7.5% as transgender women. Two-thirds of participants identified as a sexual minority person.

Qualitative findings

Innovation domains

The innovation domains refer to the Our Plan intervention as a potential program to be implemented within organizations. The most prominent CFIR domains were related to innovation: (I) relative advantage, (II) adaptability and design, and (III) desired evidence base.

Relative advantage

Relative advantage refers to the degree to which Our Plan is better than other available programs or current practices within organizations. Most participants highlighted several core benefits of Our Plan. Broadly, they perceived these advantages to include (I) increase capacity to support couples, (II) synergy with existing programs, and (III) increase reach and engagement with patients.

One of the most prominent advantages of Our Plan described by participants was that it would allow organizations to better support couples (see Table 2). Most organizations offered various degrees of sexual health services, ranging from primary care settings with limited sexual health services to ASOs and lesbian, gay, bisexual, transgender, and queer/questioning (LGBTQ+) serving CBOs that offered specialized programming for HIV prevention-care. Participants in primary care settings who offered limited sexual health services described Our Plan as an important program to promote general conversations around sexual health, as well as engage in tailored discussions about patients’ sexual health needs (e.g., viral suppression, exploring HIV prevention options).

“This topic is not really being hit on by PCP appointments [...]. I think that the whole concept to arriving in a treatment plan would be an advantage and help them improve their sexual health behaviors and also start considering how their behaviors are impacting others. Because everyone is HIV positive and not everyone’s viral loads are under control.” (23-year-old bisexual Black man, volunteer fellow in the South)

ASOs and CBOs also described how Our Plan would complement their existing services. Specifically, participants said that Our Plan could increase the capacity of other existing services, namely HIV testing and prevention programs, by addressing gaps in programming for couples. Notably, the majority of participants from these organizations said that they did not offer services for couples, except a few organizations did provide CHTC in limited contexts, such as at outreach events. In many instances, participants described not having the skills or training to provide relationship-specific support and considered Our Plan as a useful tool for directly supporting patients experiencing relationship issues.

“It’s something I’m not good at as a case manager, I’m not like trained in that aspect of like... I don’t do a lot of prevention work, and I’m not trained on how to help maintain healthy relationships, I’m not like a therapist or anything, and I don’t think it’s an alternative to therapy, but I do think it would help meet their needs because a lot of them can’t afford certain access to services.” (22-year-old bisexual Black woman; case manager in the Midwest)

Second, Our Plan was described as a promising resource to address gaps in existing services. Participants viewed Our Plan as salient example of how organizations could move away from individualized models of sexual health by engaging patients in relationships, identifying their health needs, and providing tailored support to couples. Participants also appreciated Our Plan’s sex-positive, collaborative approach to addressing couples HIV prevention needs. Several participants affirmed the importance of combining pleasure and prevention into sex education and felt that the sexual health content, in addition to educational resources, would reduce stigma and promote healthy communication among patients. Similarly, participants also appreciated how Our Plan could foster positive relationship outcomes, commenting that the relationship education modules and activities could provide couples the tools and opportunities to have healthy conversations about a variety of topics involving sex, prevention options, and boundaries.

“I like that aspect of like, hey it makes those negotiations of what we both wanna do to keep ourselves and each other safe, I love the... I like that approach of like, Oh, we aren’t really on the same page, how do we get to a place where maybe we come together and we have a plan that feels comfortable for both of us, I like that aspect of that. Because those negotiations and coaching clients on how to have those conversations is definitely a part of what we do.” (52-year-old White genderqueer person; manager in Midwest)

Finally, participants described a range of benefits that Our Plan could provide to their organization in terms of increasing patient engagement. They described Our Plan as a useful tool to help improve the quality of care, reduce stigma, and reach a broader patient population. Participants from organizations with limited digital infrastructure indicated that several Our Plan features would give staff more opportunities to stay in contact with patients and ensure the likelihood of sustained engagement in services. The most relevant features for engagement were short message service (SMS)-messaging between staff and patients and in-app referrals to clinic services (e.g., PrEP counseling, mental health counseling).

“You know, it might increase actual activity in the clinic because it seems like you can communicate with your clinic via text and get faster responses. It would be cool to schedule appointments. I think you would actually get more clientele if you can go on there, talk to your PrEP, a navigator or... And schedule appointments. That’s better than sitting on hold.” (37-year-old bisexual Black woman; testing counselor in Southwest)

Participants discussed how Our Plan could improve quality of care by addressing challenges related to limited staffing and resources. Many participants mentioned that limited staffing and resources present challenges (e.g., high turnover rates) that make providing comprehensive sexual health services difficult, such as not having the time to answer patients’ questions or disrupting continuity in treatment plans.

“Well, I guess instead, some organizations have a higher turnover rate or if their staff... And so if this helps offset the load of the existing counselors or navigators, then I think that can really help the organization improve their quality of work and help clients execute the personalized plans that we create for them or that we create together, I should say. I think other organizations’ capacity would improve or would increase.” (22-year-old queer Mestizo woman; health educator in the South)

Some participants also described how Our Plan could help overcome stigmas (e.g., sex negativity, homophobia) and privacy concerns that often negatively impacts many patients’ willingness to engage in conversations about sexual health.

“So I tell our patients who especially have had a lot of issues with stigma and self-stigma and they don’t wanna... Their experience with initially interacting with HIV care was pretty traumatic, that we obviously want you to feel better about accessing medical care, but if you just really don’t wanna talk to people, this is a sweet way to do it ’cause you can get what you need, you can get the medical care and you’re not forced into... You don’t have to deal with the anxiety of, “I gotta call my doctor.” I’d rather you message them and get it done, than sit there for two months and fret over calling.” (38-year-old bisexual White woman; program coordinator in the South)

One participant described how Our Plan could offer some degree of respite for patients who might not feel comfortable talking about sex with a provider directly. Other participants mentioned that Our Plan is a positive step towards destigmatizing sex and centering patient’s agency in their own healthcare.

“I think is a cool thing and getting to go at their own pace. And maybe also if it’s something they... As much as in the role that I am, I want people to feel very open talking about their sexual health and to destigmatize it. I also understand that especially when it’s between partners they’d want that to maybe be more private, so I think the fact that it is, again, they have the agency to do it on their own with their partners and not to have us mediating or whatever.” (23-year-old straight mixed-race woman, volunteer in the South)

When it comes to organizational reach, Our Plan represented an opportunity to attract a variety of patient populations. Most participants mentioned that Our Plan would entice younger patients who prefer or are comfortable using online platforms in their daily lives. Patients described as computer-literate were considered another audience interested in the accessibility and convenience of a mobile app or digital health interface. Patients facing transportation challenges (e.g., poor infrastructure, safety, distance to clinic) would similarly benefit from digital health services.

However, participants also described several disadvantages they perceived about the potential use of Our Plan, such as misalignment with the organization’s values in the provision of patient-centered models of care, and low interest from important priority patient populations. The most prominent disadvantages of Our Plan related to the value of in-person interactions for provider-patient relationships, especially in the context of organizations serving populations facing structural vulnerabilities of housing and/or economic instability. Participants cited multiple examples of how patients experiencing structural vulnerabilities would not be able to consistently access and potentially benefit from Our Plan due to inconsistent smartphone access, a lack of privacy in their current housing, or low computer literacy, as well as needing to attend to more pressing needs in their lives.

“Some people don’t have that flexibility [in regards to privacy to complete Our Plan]. Their kids are there or their partner are there, everyone’s working from home, they don’t have a space where they can maybe go and talk about the thing that they need to talk about in private.” (46-year-old queer White woman, organization president in the Midwest)

Many participants described how their organization preferred patient-centered, in-person interactions to adequately support patients experiencing structural vulnerabilities. Participants discussed the importance of fostering in-person connections and how this connection was integral to fostering a successful provider-patient relationship.

Innovation adaptability

Participants gave numerous suggestions for how Our Plan would need to be adapted and tailored to fit the organization’s local contexts or needs. We found three sub themes with adaptability, which included: (I) robustness of support offered through Our Plan, (II) cultural tailoring, and (III) privacy and security.

Regarding Our Plan’s robustness, participants emphasized that Our Plan should offer additional, scaffolding features and services to address patients’ needs more holistically, especially for organizations serving patients experiencing multiple structural vulnerabilities. One of the most frequently cited desired features was a more holistic resource list. Participants offered a variety of suggestions to make Our Plan’s resource list more useful for patients with multiple, competing needs, such building out medical service resources beyond HIV/STI testing, HIV care, and education on PrEP and post-exposure prophylaxis (PEP). Participants discussed the importance of including other important resources such mental health (e.g., gender-affirming materials, counseling referrals), legal services, social services (e.g., housing, nutrition, and health insurance navigation), and sex-positive relationship resources [e.g., Bondage and Discipline/Dominance and Submission/Sadism and Masochism (BDSM) safety]. Moreover, participants wanted the ability to curate resource lists to reflect local contexts, such as including services offered by their organization or partner agencies, and to vet and edit the listed resources for quality, appropriateness, and accessibility.

“Oh, there’s these few different organizations and they all say that they do the same thing, but some of them, people tell you, “Oh, I keep calling them and it’s like a black hole.” But other ones, people are actually like ‘no’. So I mean, if they’re actually kind of tried-and-true resources that people have been able to successfully access. I think any resources... Because the app is concerned about not being stigmatizing and about being specific to the LGBTQ or MSM community. Then making sure that those resources are also people who are whatever. Those resources are also quite friendly, all those things.” (23-year-old straight White man, community health worker in the South)

Other salient examples of desired features in Our Plan to support their patients include components for direct linkage to complimenting services within the organization, such as PrEP navigation or couples therapy, a live chat feature to ask questions and practice conversations, and accountability tools (e.g., checklist for prevention plan, reminders for medical appointments or medication dosing).

“With the lack of planning skills that adolescents have and follow through skills that adolescents have, the reminders, the checklist, and the accountability, the plan is great, let’s make the plan together. And then both of us can go back into the app together and be like, “You totally didn’t go to the doctor.” And then like, “Oh yeah, I forgot.”” (38-year-old bisexual White woman; program coordinator in the South)

In regard to the organization, there was a practical need for interoperability with digital infrastructure such as electronic health records to more seamlessly integrate Our Plan into existing services (e.g., healthcare visits) and reduce staff burden. Participants also described the importance of being able to customize the admin portal dashboard to meet their organizational workflow and data and accounting reporting needs.

“I would definitely like the data tools. I think probably sharing, with you all and like building like what we actually need. And adding that... Those features to where like, perhaps I can just go in and those features are already within that administrative side that I’m just clicking for like that month or gathering data.” (35-year-old gay Black man, project coordinator in Midwest)

Participants highlighted multiple opportunities for modifying content to better reflect their patient populations’ lives and environments, or cultural tailoring Several participants wanted to be able to customize Our Plan’s educational content to include specific statistics and examples that directly acknowledge patients’ geography and cultural identities and ground their expectations, such as localizing HIV statistics to reflect relevant trends (e.g., impact of HIV on Black communities in the South) or including stories that speak to common challenges among patient populations (e.g., substance use in relationships). Some participants described the importance of attending to intersectional identities and inclusiveness (e.g., inclusiveness of transgender men, those in polyamorous relationships and sex work). Most participants underscored the importance of Our Plan to be culturally tailored to be more inclusive of diverse communities, including using more visuals of people, cultural references, and stories or examples that reflect different cultural backgrounds. In the words of one participant:

“I think people like to see others that look like themselves in these types of things especially for a Latino gay man, he... I think he would really like to see... They would like to see representation of themselves in these kinds of resources...And when you’re increasing that visibility and representation of those communities, you’re usually able to build trust much easier and so maybe they’ll be more likely to use it, because they see it as like, “Oh, this has been catered for me.” It’s not just another thing that was translated into Spanish, it’s like, “Oh, okay, I could actually use this.”” (22-year-old bisexual Mestizo woman; health educator in the South)

Relatedly, participants mentioned the importance of being able to tailor or alter the images, videos, and content of Our Plan to reflect the communities most impacted by the HIV epidemic to increase acceptability and trust amongst the communities they serve. Lastly, regarding privacy, participants identified a need for improvement to safeguard against potential security breaches and protect patients’ confidentiality, especially given the sensitivity of relationship topics among patients. While participants enjoyed Our Plan’s collaborative features, there was a concern that partners would not have control of information that would be shared with their partner. In addition, there was also a general concern about data access within and out of organizations. Some participants suggested explicit messaging about what, when, and how information will be shared with partners to avoid unintended relationship conflict or intimate partner violence.

Evidence base

Participants spoke about multiple different types of evidence that would be needed to demonstrate effectiveness and subsequently their willingness to implement Our Plan at their organization (see Table 2). Participants emphasized the need for qualitative data from end-users and organization staff and often mentioned practical evidence rather than theoretical. These types of evidence were broken down into three subthemes: (I) impact, (II) acceptability; and (III) usability and functionality.

In regards to impact, the majority of participants mentioned that they would want to know whether Our Plan was effective in increasing HIV/STI testing and reducing HIV incidence. Within this subtheme, participants often noted the importance of “success stories” demonstrating that Our Plan was effective in helping couples create and maintain a prevention plan.

“They’ll [staff] want to see what the data is around, rates for HIV and STIs. They’ll wanna see if this platform has had any impact on that at all, for the individuals using it, they’ll wanna see any success stories as a result of this platform being used.” (61-year-old gay Black man, prevention manager in Midwest)

Across the interviews, acceptability data in the form of personal testimonies from couples who completed Our Plan was the most robust and requested type of evidence that they would want. Participants also noted the importance of knowing whether Our Plan had any negative impact on relationships or contributed to intimate partner violence.

“I think this is a nightmare scenario of what could happen but it’s like... You worry. I think one thing that’s popped in my head is you never wanna think that someone is going to use this tool as a way of trying to control their partner or be invasive in their life.” (23-year-old straight White man; community health worker in the South)

Acceptability also included metrics on user engagement (e.g., completion of modules), enrollment and retention, as well as whether clients enjoyed Our Plan both in terms of content and information, as well as the online modality.

“I like the numbers [laughter] but I think patient stories are important because the numbers don’t tell everything, the numbers don’t tell about the experience as much. And I have found that in our world, the patient experience affects everything. I mean, it affects adherence, it affects retention and care and everything is affected by how the patient feels about it. The HIV world is not just straight up medical... I think that patient stories are important too. So to answer your question, a combination of both [numbers and patient stories].” (38-year-old bisexual White woman; program coordinator in the South)

Usability and functionality evidence was described as important for end-users as well implementers. Specifically, participants discussed the need to have documented evidence that demonstrates how technologically challenging or easy it is to use Our Plan from organizational staff and clients who have already used it.

“If there’s too many technical difficulties, if they just flat out say, ’cause a lot of our clients are very straightforward, “I don’t like it. What is this? It doesn’t make sense to me,” if we just get a lot of negative feedback [I wouldn’t continue using it with clients].” (29-year-old straight woman; trans health coordinator in the Midwest)

Participants also noted the importance of immediate feedback and resolutions by the Our Plan developers if problems arose and how long the wait times would be, which could aid in their reluctance to implement and use the program.

“And then on my side, if I got to... If I need to talk to someone to fix something, if I had the AT&T experience, where you’re on hold for two hours and never get a person I’m just hanging up and I’m saying, forget it. [chuckle] So whoever I need to talk to, to fix stuff, I need to be able to talk to, to fix stuff.” (38-year-old bisexual White woman; program coordinator in the South)

Across each of the “evidence-base” subthemes, participants noted the importance of transparency of reporting end-users’ demographic information, with a specific desire for evidence to come from people of color.

“Just like how much engagement is there over time on, specifically among Black, same gender loving men, because as you know yourself, a lot of times studies are not inclusive of or focused on that community.” (35-year-old gay Black man; project coordinator in the Midwest)

Inner setting

The inner setting refers to the ways in which participants could envision Our Plan being implemented within their organization. The three most prominent inner setting domains were: (I) the compatibility of Our Plan within existing services, (II) culture in determining whether Our Plan aligned with the values of the organization, and (III) the required materials and equipment to implement Our Plan.

Compatibility

Compatibility refers to the degree to which Our Plan fits within workflows, systems, and processes at implementing organizations. In general, participants described different types of programs and opportunities to integrate Our Plan within their current programming but also noted potential facilitators and barriers to integrating Our Plan within their organization. The majority of participants discussed how they could imagine Our Plan fitting into the existing HIV/STI prevention services, including testing and PrEP delivery or within primary care service. Others suggested that Our Plan could fit within behavioral services, within a research division, and/or outreach services and events. One participant mentioned that Our Plan would be best implemented during front desk interactions.

As shown in Table 3, participants reported several facilitators or benefits to implementing Our Plan within their organization. Participants described three primary benefits to Our Plan, specifically serving as a means to help couples get engaged in care and take ownership of their health, facilitating conversations between partners about their sexual health, and improving care coordination and retention in care. Regarding care coordination, one participant described how Our Plan may even be beneficial to support their clients experiencing intimate partner violence.

“Right. Or like, have you felt safe in your relationship, going through all these topics? If not like, how can we support you in that? Like maybe if there’s an interest in therapy for folks, having those resources available with their social workers and having to look ’em up blind but like, you know, people decide they wanna work on couples therapy or if there’s new concern for... In a part, whether personal violence, like how do we, again, support clients who might be going through things they haven’t previously.” (33-year-old straight Black/mixed race woman; medical provider in the Northeast)

The primary barrier or concern reported was “losing human touch”, as some participants described how their organization had a strong preference for in-person interactions with their clients. Specifically, several participants described how their clients found online interventions without a human connection to feel “cold”, “impersonal”, and “intrusive”.

“I just don’t think for our clientele, I’m just being honest, I think for our clients, our patients, it’s a little too intrusive. It’s a little... And not only is it intrusive, it’s very impersonal. You know what I mean? It’s like people want a human being, they want that human contact.” (55-year-old gay White man; health educator in Midwest)

Other barriers included that the intake process was already burdensome and offering a DHI may be intimidating for their clients, leading implementers to be reluctant to offer Our Plan to their patients.

Culture

Culture refers to the degree to which participants perceive that Our Plan is aligned with the values, beliefs, and norms within their organization. Participants explicitly mentioned that Our Plan fit in their organizational culture and specifically described their values around fostering communication, including communication with partners.

“We really value communicating. So I think this is just an extension of communicating and it would teach our clients to like communicating with each other and like their partners and stuff like that. So I think it fits right into our culture so yeah.” (22-year-old straight Black woman; HIV tester and outreach coordinator in the South)

Participants also mentioned how the sex-positive and de-stigmatizing framing of Our Plan was aligned with their current or aspirational organizational values.

“So I think we’re actually at a really cool place right now where a lot of folks who are not interested in sex positivity and thinking about pleasure and thinking about people as people having sex and interactions with one another have left the field because they’ve realized they don’t have that value. We’ve now been welcoming more people and we’re explicitly asking about that. So I think right now we’re building a culture where that’s a foundation. Everybody that gets hired is learning about sex positivity and pleasure and how we talk to people about sex.” (30-year-old White nonbinary person; clinic manager in the Midwest)

In addition, a few participants explicitly mentioned their organization valued trying new programs for their clients.

“I think as an organization at large, there’s a receptiveness to this sort of wanting to be innovative and on the cutting edge of service delivery, so I think that would be advantageous to getting people to take this up and wanna offer it to clients. I’m trying to think if... I really don’t think there’d be many people pushing back, just because, again, we’ve seen what it means to open doors for people.” (33-year-old straight mixed-race woman; medical provider in Northeast)

Barriers or misalignments between organizational cultures and Our Plan varied. Participants noted that there could be a potential lack of interest in couples-based programming, particularly among staff who might be implementing it, coupled with limited education in harm reduction strategies, discomfort in discussing sexual health, limited literacy about and using technology, as well as high staff turnover with concerns about who would train new staff. There was also concern that staff implementing Our Plan may not taking the time to fully understand their client’s needs before suggesting the program, resulting in unintentionally or inadvertently profile clients for the program based on their own misperceptions and views towards individuals. One participant captures this idea and emphasizes the importance of checking in beforehand to gauge if and how using an DHI, like Our Plan, could help meet their client’s needs.

“Having people that work in this field and still holding onto bias and misconceptions around queer people and trans people and all the things within that. And I think if I’m offering an app, it’ll just be like, Oh, okay, you are Black and gay or you’re this or that. Or you came in here, this... You know, you clearly need this. Versus, we have this app. How do you feel? What do you need, would this work for you? I think just doing more of that and not so much of this because of my own perceived misconceptions and ideologies, you clearly need this. And I think that’s like... That’s even with condoms, like we’re constantly throwing condoms at people. And, even for me, when I first came out, it was like, ‘you’re gay, you are gonna get HIV.’” (35-year-old gay Black man; project coordinator in Midwest)

Some participants also described how their organization valued in-person interactions. This was particularly evident among organizations that served patients who were experiencing structural vulnerabilities. For example, one participant described how their patients often experience homelessness without consistent access to phones or the internet, which would require them to come into the organization to engage in the program.

“Yeah, so we do have a great number of clients experiencing homelessness that lose their phones all the time or their phones get turned off, and don’t have access to the internet. So that would be a barrier with implementation for those clients. So it may not be the most feasible thing for them to do unless they come in and do it on a computer since it’s web based.” (32-year-old queer Black man; administrator in the West)

Materials and equipment

The materials and equipment domain refers to the supplies that participants described as needing to be available to implement and deliver Our Plan. Participants described the need for staffing and training resources to implement any DHI, and specifically Our Plan. Given staffing shortages and high staff turnover, participants discussed how the implementation of Our Plan would require additional funding for staff to support end-users, along with technological support.

“I think that there would have to be some sort of funding that is paired with an intervention like this. If this is going to be used for a larger study and as something to prove that it is worthwhile, then we should also get some support in terms of technical assistance in using it, in terms of funding a part-time person to be the person who is the manager of this intervention if it really is going to be implemented in a major way.” (30-year-old White nonbinary person; clinic manager in Midwest)

Participants also discussed the need for a relationship with the Our Plan developers that demonstrates developers’ investment in the communities that the organizations serve beyond academic accolades.

“I think something I have recently somewhat seen just disconnects in the academic world, are you kinda losing some of that personal touch when some people, like the white-collar world, do not see with the blue collar world. So just make sure you make that connection in a meaningful, truthful and honest way that maybe the person promoting it is also someone that uses it or someone that has a partner that they also say they would benefit from if they had used it when they were younger or something like that. Giving a personal story to show the efficacy of it.” (25-year-old gay Asian man; study coordinator in the South)

The majority of participants mentioned training as a necessary and needed resource to include for the implementation of Our Plan. Training included several different components (e.g., how to implement Our Plan, couples’ communication skills, LGBTQ+ and sexual health promotion cultural competency), that were specific to Our Plan but also spanned other topics and skills relative to offering services to couples and partnered individuals. Participants also mentioned the need for ongoing and immediate support and connection with the developers as questions or challenges arose during implementation.

“I mean, I think definitely there would need to be some type of, opportunity for the training so to say, you know what I mean? So that it’s not just something that’s just thrown at us and like, ‘Hey, here, you go figure it out. Figure it out if you want to.’ But, so I think that’s really helpful. Just any type of some... Making sure that there is support. So that if questions on our end, like the back office type end come up, that it’s there, they’re okay. We can reach out here and it’s not gonna take three weeks for this question to be answered. And, you know what I mean?” (43-year-old mixed race trans woman; prevention coordinator in the Midwest)

Discussion

Findings from the present study provide important and useful information about implementation determinants of Our Plan—a couples-based DHI designed for new relationships of same-gender male couples and dyadic, sexual partnerships. ASOs and CBOs play a critical role in addressing HIV inequities, and DHIs have the potential to help improve client reach, provide additional educational information, and help complement the services offered at these organizations. Yet, what is required to make this a reality—that is, involving and/or integrating DHIs with ASOs and CBOs for HIV prevention-care—requires careful attention and consideration. Our findings relate to one example of DHI and what might be needed to bridge the gap of ‘virtual’ with ‘in-person’ services to subsequently expand and enhance HIV prevention-care in the US.

Overall, potential organization implementers thought Our Plan could be one way to address service and programming gaps via a couples-based DHI, which aligned with their organizations’ mission and culture to provide HIV prevention and care in a holistic manner. Given documented barriers to providing in-person services for couples, such as CHTC (e.g., limited staffing and challenges with service integration) (12), most potential implementers were excited about the potential to offer a couples-based DHI that focused on communication skills with sex positive framing, and saw the benefits of increasing their capacity to support couples. Importantly, participants discussed several plausible configurations for integrating Our Plan into existing workflows that could bolster organizational capacity (e.g., increased referrals, improved retention, reducing staff effort). Their thoughts and excitement collectively confirmed that efforts to develop couples-based and couples-focused DHIs for sexual minority male couple/dyad groups have and will continue to be central to HIV prevention and treatment efforts writ large (6,8-10).

It is important to note that insights and feedback from potential implementers may have stemmed from their limited exposure to Our Plan (i.e., very brief videos providing an overview of the DHI), including some of the recommendations they had about what would be needed for their organization to successfully use this DHI. Nonetheless, potential implementers discussed the need for Our Plan to offer sufficient safeguards to protect their clients’ privacy and safety, specifically around information sharing in the relationship and partners working together as a couple/dyad. Their concern for future clients’ safety is warranted. The videos showcasing Our Plan did not include two important aspects about it: (I) the data security features that were used to host and access it, and (II) the informed consent and onboarding process that each partner/couple had undergone before experiencing the intervention within the DHI.

As noted in the Introduction, Our Plan used several data security features for both types of end users (participants/clients, admin/research team members), as well as protocols to ensure HIPAA compliance. For brevity purposes, the videos did not showcase these details about Our Plan and as a result, potential implementers would not have known about these features. It is recommended that these important data security features of a DHI be highlighted to potential implementers, including how the features work and what purpose they serve.

Regarding clients’ safety, the consent process entails that each individual currently feels safe in the relationship, has not experienced any type of emotional, sexual, mental, or physical harm from their current relationship partner (ever or recent), and does not feel coerced to participate in the study or use Our Plan. The feedback from the potential implementers highlights the need to provide this critical information when showcasing and describing DHIs regarding scale up, as well as thinking about what additional safeguards might be needed outside of the research study context. One possibility would be to maintain the aforementioned research informed consent process for Our Plan (and similar DHIs for dyads) when offered in the real world context. ASOs and CBOs could also consider integrating and offering their own consent or informational awareness process to their clients for DHI(s) they intend to offer or refer to for HIV prevention and care. Another option to consider for client safety could entail updating core behavioral mechanisms embedded within Our Plan that encourage couples to work together to create an HIV prevention-care plan that meets both partner’s needs. The change would include incorporating design features to give clients control over what and how information is shared with their partner to avoid unintended adverse events such as relationship conflict or intimate partner violence. As such, the DHI—pending on users’ preferences—may result in being a couples-focused or relationship-oriented DHI rather than a true couples-based DHI. The drawback of offering this design feature is that it changes the core principle of providing couples-based interventions to couples and sexual dyads where the overarching goal is to help partners be on the ‘same page’ about their sexual health.

Potential implementers also underscored the importance for Our Plan to better reflect the identities (i.e., through images) and needs [i.e., content with related resources (BDSM)] of the local communities served by their implementing CBO or ASO. Human-centered design principles and members of the community of focus were used to guide and provide input toward the development of Our Plan, including its current iteration. Despite these efforts and in light of what potential implementers shared, the images in the DHI lacked sufficient representation of all community types served by the potential implementing CBOs and ASOs, nor comprehensive content with relevant resources. Cultural tailoring and providing other major updates to the base form of a DHI has long been noted as a necessity as well as a significant challenge (28). The challenge of DHIs to meet the needs of varying communities by CBO/ASO is centered on providing a design that allows flexibility to ‘add’, ‘remove’, and/or ‘update’ images, content, and resources for cultural tailoring purposes, and whether needs of varying communities can be met from the same DHI. A related challenge centers on who or what entity would hold responsibility and needed resources to change and/or update a DHI for any given CBO or ASOs. Another approach would entail involving potential implementers in the development of the DHI from the onset (i.e., bottom-up) (29,30), which may aid in creating a practical design that permits more flexibility to tailor and allow sufficient representation of communities served by the implementing CBO and ASOs.

From an organizational-level perspective, there were also perceived barriers to implementing Our Plan. Consistent with prior findings (23,24), potential implementers discussed how their organization had experienced difficulties switching to virtual services during the COVID-19 pandemic. These potential implementers had concerns that Our Plan would not complement their organization’s holistic, person-centered models of care and work within their culture, primarily because of the necessity for in-person connections with clients. From their perspective, many of their clients experience significant structural vulnerabilities, and may not have the technology literacy or data plans necessary to use a DHI such as Our Plan. In the U.S., it has been estimated that approximately 21 million people do not have fixed broadband internet access and rely on mobile devices for online access (31), which significantly limits digital connectivity (32). For example evidence suggests that 25% of Latinx and 23% of Black individuals are solely reliant on smartphones for internet access compared to 12% of White individuals (33). And while many people living in poverty have some form of internet access (34), individuals with lower income levels are more likely to solely rely on smartphones compared to those with higher income-levels who also have access to broadband internet connections (35). Notably, these patterns occur at the neighborhood level whereby evidence suggest that neighborhoods with low-level incomes have the lowest internet subscription rates (36) and nearly half of low-income households rely on limited cellphone data plans or public wifi hotspots for online access (37). While DHI have the potential to increase access to health services by seemingly addressing a host of social determinants of health in low-income and rural communities, disparities in digital connectivity represent a fundamental challenge to the ensuring that the benefits of DHI like Our Plan are equitably distributed and accessed among communities that might benefit from it most. The COVID-19 pandemic confirmed research about digital divides among populations experiencing structural vulnerabilities among organizations that them as well (38). Though many organizations had adopted DHIs, specifically telehealth, to maintain continuity of care-services during the COVID-19 pandemic, research has noted that many of the organizations who serve communities most heavily impacted by the HIV epidemic have their own set of challenges for digital access, often lacking technological literacy and resources (i.e., digital connectivity) to effectively use DHIs (38). Organizational implementers emphasized the importance and need to form strong partnerships between developers/researchers and stakeholders (e.g., implementing staff, end-users) throughout the entire research-to-implementation process to help ensure the usability, uptake, and impact of Our Plan, including among populations experiencing structural vulnerabilities (39).

Regardless of the organizational culture and client population, potential implementers noted that in order to integrate Our Plan into their organization’s existing services, a few requirements would need to be met. First, the interoperability of Our Plan’s platform must be able to co-exist or ‘talk’ with the CBO/ASO’s information systems used for electronic health records and case management, for example. Ongoing technological training and support for staff and end-users would also be needed amongst potential implementation organizations. Lastly, potential implementers expressed the potential that Our Plan could be billed through existing federal funding sources (e.g., the Ryan White HIV/AIDS Program). The aforementioned practicalities have previously been documented about the integration of DHIs within healthcare settings (40,41), and are particularly important for smaller organizations with limited resources (42).

Taken together, our findings highlight the potential for Our Plan, as a couples-based DHI by potential implementers in community settings. Study findings also highlighted importance of utilizing Health Equity Implementation Frameworks in the development-implementation process (43-45). Implementation determinants were often intertwined with principles of social justice. As such, Our Plan and other DHIs must account for different the cultural factors related to the implementers and end-users, potential bias in clinical/service encounters, and the larger social context of the organization that are critical for implementation success.

Limitations

The current project is among one of the first couples-based HIV prevention DHIs to inquire about the determinants of implementation during the developmental phase. The findings from this work must be considered in light of the project’s limitations. Although we were able to reach several organizations across different EHE jurisdictions, we were unable to reach all jurisdictions, thereby limiting generalizability of the findings. In addition, participants were from organizations that our team had personal connections with or were willing to participate in the project. Interviewers were selected who were not part of the development of Our Plan or had any investment in its success, which allowed participants to openly share their opinions. Nonetheless, social desirability bias may be possible. We chose individual interviews to learn about potential implementers unique perspectives about Our Plan within their organizational context. Future evaluation research may consider using focus groups to generate a shared understanding of implementation determinants of Our Plan. Our Plan currently does not include specific accessibility features; however, the next iteration will include accessibility features to improve experiences for people with disabilities, which include careful consideration of color choices and audio. Finally, we inquired about hypothetical implementation where participants only saw a brief overview of Our Plan without having full access to or the ability to use the DHI. Thus, study findings may reflect this limited exposure to the DHI and may differ compared to potential implementers being able to use Our Plan as both types of users (i.e., client/participant couple vs. CBO/ASO staff).

Conclusions

In an era in which DHIs, including couples-based DHIs, continue to be developed to address HIV prevention and care continua outcomes, our findings highlight the importance of working closely with potential implementers and end-users throughout development phases and planning early for sustainable scale up. We found high interest in Our Plan from potential implementers to address gaps in service needs for couples. Many potential implementers saw value in Our Plan as a supplement to existing in-person services. Addressing barriers regarding cultural, financial, and socio-structural realities of implementing organizations and the populations they serve is critical to the success of adoption and potential scalability of Our Plan. Taken together, our findings underscore the importance of including principles of implementation-dissemination early on and throughout the research development process to help ensure the uptake and sustainability of couples-based DHIs, such as Our Plan.

Acknowledgments

The authors would like to thank the individuals who participated in this study and C.P. Ku for their assistance with de-identifying and verifying the accuracy of the transcripts.

Funding: This study was funded by a grant from the National Institutes of Health (No. 2U54MD012393-05S2).

Footnote

Data Sharing Statement: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-57/dss

Peer Review File: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-57/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-57/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. All participants provided informed consent to participate and to have their respective interview audio recorded, transcribed, checked for accuracy, and de-identified for research purposes. The study design and associated procedures were approved by Florida International University Institutional Review Board (No. IRB-21-0485). The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013).

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Centers for Disease Control and Prevention. Diagnoses of HIV infection in the United States and dependent areas, 2019. HIV Surveillance Report. 2021. Available online: https://www.cdc.gov/hiv/library/reports/hiv-surveillance.html
2. Goodreau SM, Carnegie NB, Vittinghoff E, et al. What drives the US and Peruvian HIV epidemics in men who have sex with men (MSM)? PLoS One 2012;7:e50522. [Crossref] [PubMed]
3. Sullivan PS, Salazar L, Buchbinder S, et al. Estimating the proportion of HIV transmissions from main sex partners among men who have sex with men in five US cities. AIDS 2009;23:1153-62. [Crossref] [PubMed]
4. Burton J, Darbes LA, Operario D. Couples-focused behavioral interventions for prevention of HIV: systematic review of the state of evidence. AIDS Behav 2010;14:1-10. [Crossref] [PubMed]
5. Martinez O, Fernandez MI, Yang C, et al. Feasibility and Acceptability Pilot Test of Connecting Latinos en Parejas: A Couples-Based HIV Prevention Intervention for Latino Male Couples. Am J Mens Health 2023;17:15579883231167106. [Crossref] [PubMed]
6. Mitchell JW, Lee JY, Wu Y, et al. Feasibility and Acceptability of an Electronic Health HIV Prevention Toolkit Intervention With Concordant HIV-Negative, Same-Sex Male Couples on Sexual Agreement Outcomes: Pilot Randomized Controlled Trial. JMIR Form Res 2020;4:e16807. [Crossref] [PubMed]
7. Starks TJ, Robles G, Doyle KM, et al. Motivational interviewing with male couples to reduce substance use and HIV risk: Manifestations of partner discord and strategies for facilitating dyadic functioning. Psychotherapy (Chic) 2020;57:58-67. [Crossref] [PubMed]
8. Stephenson R, Sullivan SP, Mitchell JW, et al. Efficacy of a Telehealth Delivered Couples' HIV Counseling and Testing (CHTC) Intervention to Improve Formation and Adherence to Safer Sexual Agreements Among Male Couples in the US: Results from a Randomized Control Trial. AIDS Behav 2022;26:2813-24. [Crossref] [PubMed]
9. Gamarel KE, Darbes LA, Wall KM, et al. A Relationship-Focused HIV Prevention Intervention for Young Sexual Minority Men in the United States: A Pilot Randomized Controlled Trial of the We Prevent Intervention. AIDS Behav 2023;27:2703-19. [Crossref] [PubMed]
10. Newcomb ME, Swann G, Macapagal K, et al. Biomedical and behavioral outcomes of 2GETHER: A randomized controlled trial of a telehealth HIV prevention program for young male couples. J Consult Clin Psychol 2023;91:505-20. [Crossref] [PubMed]
11. Starks TJ, Dellucci TV, Gupta S, et al. A Pilot Randomized Trial of Intervention Components Addressing Drug Use in Couples HIV Testing and Counseling (CHTC) with Male Couples. AIDS Behav 2019;23:2407-20. [Crossref] [PubMed]
12. Stephenson R, Grabbe KL, Sidibe T, et al. Technical Assistance Needs for Successful Implementation of Couples HIV Testing and Counseling (CHTC) Intervention for Male Couples at US HIV Testing Sites. AIDS Behav 2016;20:841-7. [Crossref] [PubMed]
13. Nelson KM, Perry NS, Horvath KJ, et al. A systematic review of mHealth interventions for HIV prevention and treatment among gay, bisexual, and other men who have sex with men. Transl Behav Med 2020;10:1211-20. [Crossref] [PubMed]
14. Gamarel KE, Durst A, Zelaya DG, et al. ReACH2Gether: Iterative Development of a Couples-Based Intervention to Reduce Alcohol use Among Sexual Minority Men Living with HIV and Their Partners. AIDS Behav 2024;28:1244-56. [Crossref] [PubMed]
15. Schueller SM, Muñoz RF, Mohr DC. Realizing the Potential of Behavioral Intervention Technologies. Curr Dir Psychol Sci 2013;22:478-83. [Crossref]
16. Sullivan PS, Jones J, Kishore N, et al. The Roles of Technology in Primary HIV Prevention for Men Who Have Sex with Men. Curr HIV/AIDS Rep 2015;12:481-8. [Crossref] [PubMed]
17. Mitchell JW, Gamarel KE, Kam KK, et al. Perceived facilitators of decision-making and usage of biomedical HIV prevention strategies: findings from an online, qualitative study with same-gender male couples in the United States. AIDS Care 2021;33:1209-17. [Crossref] [PubMed]
18. Mitchell JW, Wu Y, Gamarel KE. Timing and Sequence of Events: HIV Status Disclosure, Condomless Anal Sex, and Sexual Agreement Formation Among Same-Sex Male Couples. J Sex Res 2019;56:728-39. [Crossref] [PubMed]
19. Pettit AC, Pichon LC, Ahonkhai AA, et al. Comprehensive Process Mapping and Qualitative Interviews to Inform Implementation of Rapid Linkage to HIV Care Programs in a Mid-Sized Urban Setting in the Southern United States. J Acquir Immune Defic Syndr 2022;90:S56-64. [Crossref] [PubMed]
20. Bowleg L, Malekzadeh AN, Mbaba M, et al. Ending the HIV epidemic for all, not just some: structural racism as a fundamental but overlooked social-structural determinant of the US HIV epidemic. Curr Opin HIV AIDS 2022;17:40-5. [Crossref] [PubMed]
21. Robillard AG, Julious CH, Smallwood SW, et al. Structural Inequities, HIV Community-Based Organizations, and the End of the HIV Epidemic. Am J Public Health 2022;112:417-25. [Crossref] [PubMed]
22. Wilson MG, Lavis JN, Guta A. Community-based organizations in the health sector: a scoping review. Health Res Policy Syst 2012;10:36. [Crossref] [PubMed]
23. Labisi T, Regan N, Davis P, et al. HIV Care Meets Telehealth: a Review of Successes, Disparities, and Unresolved Challenges. Curr HIV/AIDS Rep 2022;19:446-53. [Crossref] [PubMed]
24. Yelverton V, Qiao S, Weissman S, et al. Telehealth for HIV Care Services in South Carolina: Utilization, Barriers, and Promotion Strategies During the COVID-19 Pandemic. AIDS Behav 2021;25:3909-21. [Crossref] [PubMed]
25. Damschroder LJ, Reardon CM, Opra Widerquist MA, et al. Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR Outcomes Addendum. Implement Sci 2022;17:7. [Crossref] [PubMed]
26. Damschroder LJ, Aron DC, Keith RE, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50. [Crossref] [PubMed]
27. Clarke V, Braun V, Hayfield N. Thematic Analysis. In: Smith JA. editor. Qualitative Psychology: A Practical Guide to Research Methods. London: SAGE Publications, 2015:222-48.
28. Li DH, Brown CH, Gallo C, et al. Design Considerations for Implementing eHealth Behavioral Interventions for HIV Prevention in Evolving Sociotechnical Landscapes. Curr HIV/AIDS Rep 2019;16:335-48. [Crossref] [PubMed]
29. Lyon AR, Brewer SK, Areán PA. Leveraging human-centered design to implement modern psychological science: Return on an early investment. Am Psychol 2020;75:1067-79. [Crossref] [PubMed]
30. Stiles-Shields C, Cummings C, Montague E, et al. A Call to Action: Using and Extending Human-Centered Design Methodologies to Improve Mental and Behavioral Health Equity. Front Digit Health 2022;4:848052. [Crossref] [PubMed]
31. Horrigan JB. Chapter 1: Who Uses Libraries and What They do at Their Libraries. 2015. Available online: https://www.pewresearch.org/internet/2015/09/15/who-uses-libraries-and-what-they-do-at-their-libraries/
32. Sanders CK, Scanlon E. The Digital Divide Is a Human Rights Issue: Advancing Social Inclusion Through Social Work Advocacy. J Hum Rights Soc Work 2021;6:130-43. [Crossref] [PubMed]
33. Nouri SS, Khoong EC, Lyles CR, et al. Addressing Equity in Telemedicine for Chronic Disease Management During the Covid-19 Pandemic. NEJM Catalyst 2020;1:1-13. [Crossref]
34. Opportunity for all? Technology and learning in lower-income families. The Joan Ganz Cooney Center at Sesame Workshop. 2016. Available online: https://www.joanganzcooneycenter.org/wp-content/uploads/2016/01/jgcc_opportunityforall.pdf
35. Mobile Fact Sheet. 2019. Available online: https://www.pewresearch.org/internet/fact-sheet/mobile/
36. Signs of digital distress: mapping broadband availability and subscription in American neighborhoods. The Brookings Institution, Metropolitan Policy Program. 2017. Available online: https://www.brookings.edu/articles/signs-of-digital-distress-mapping-broadband-availability/
37. Sieck CJ, Sheon A, Ancker JS, et al. Digital inclusion as a social determinant of health. NPJ Digit Med 2021;4:52. [Crossref] [PubMed]
38. The Effects of the COVID-19 Pandemic on Organizations in the Deep South. Center for Health Policy and Inequalities Research, Duke University. 2021 [cited September 4, 2023]. Available online: https://southernaidscoalition.org/wp-content/uploads/2022/01/SAC-COVID19Report-Jan28.pdf
39. HIV by Group. 2022 [cited August 23, 2023]. Available online: https://www.cdc.gov/hiv/group/index.html
40. Klein GO. Standardization of health informatics - results and challenges. Yearb Med Inform 2002;103-14. [PubMed]
41. Bally ELS, Cesuroglu T. Toward Integration of mHealth in Primary Care in the Netherlands: A Qualitative Analysis of Stakeholder Perspectives. Front Public Health 2020;7:407. [Crossref] [PubMed]
42. Anuyah O, Badillo-Urquiola K, Metoyer R. Characterizing the Technology Needs of Vulnerable Populations for Participation in Research and Design by Adopting Maslow’s Hierarchy of Needs. CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Hamburg, Germany, 2023.
43. Fortuna KL, Kadakia A, Cosco TD, et al. Guidelines to Establish an Equitable Mobile Health Ecosystem. Psychiatr Serv 2023;74:393-400. [Crossref] [PubMed]
44. Woodward EN, Singh RS, Ndebele-Ngwenya P, et al. A more practical guide to incorporating health equity domains in implementation determinant frameworks. Implement Sci Commun 2021;2:61. [Crossref] [PubMed]
45. Woodward EN, Matthieu MM, Uchendu US, et al. The health equity implementation framework: proposal and preliminary study of hepatitis C virus treatment. Implement Sci 2019;14:26. [Crossref] [PubMed]