Video selection and assessment for an app-based HIV prevention messaging intervention: formative research

Introduction

In the United States (US), gay, bisexual, and other men who have sex with men (GBMSM) are overrepresented in human immunodeficiency virus (HIV) infection, accounting for 69% of new HIV diagnoses in 2018 (1), and need new prevention modalities. Because of the high acceptability of electronic health (eHealth) interventions among GBMSM (2-5), reaching men via smartphones with engaging HIV/STI prevention media can provide a comprehensive, scalable approach to culturally appropriate and accurate health information (6-8).

Videos are an effective way to deliver HIV/STI prevention messaging to GBMSM (9-14) and can convey user-friendly scientific information more effectively (e.g., through modeling of behaviors and storytelling) than conventional text or graphics found in web-based or print materials (15,16). Viewing videos has been associated with reductions in HIV/STI risk among at-risk populations (6,17-20), including GBMSM (7,10,12,20). A review of video-based interventions found that video-delivered health information was effective at modifying certain behaviors including HIV testing and treatment adherence (18).

Mobile technology is an efficient and accessible approach to disseminating HIV/STI prevention information to GBMSM: 84% of US males own a smartphone (8). The high population coverage of mobile phones translates to a broad reach of messaging at minimal cost (21), and information can be viewed privately (20). Some social media interventions for GBMSM vulnerable to HIV acquisition have been efficacious, in part due to the inclusion of videos, interactive games, mobile applications (apps), online communities, and text messaging (14,22,23).

Due to the sheer volume of publicly available videos, selecting relevant and scientifically accurate content is challenging. We describe the process of reviewing and identifying potential videos, conducting in-person interviews with GBMSM to determine the acceptability and comprehension of the videos, and selecting a final set of publicly available videos for an app-based, HIV/STI prevention messaging randomized controlled trial (RCT), Mobile Messaging for Men (M-Cubed). The M-Cubed app was designed to deliver brief written and video-based sexual health prevention messages and link participants to prevention and healthcare resources, such as pre-exposure prophylaxis (PrEP), HIV testing, and HIV care locations (24). A full description of the M-Cubed app, study design, and intervention is reported elsewhere (24).

Methods

Video iterative review process

In 2016, we compiled a list of CDC-made, public service announcements and conducted an internet search of publicly available videos using the keywords “HIV Prevention” and filtering for video content. This resulted in an initial list of 155 videos, or approximately seven hours of video footage to review, that addressed HIV prevention topics in one of six domains (ART, Condoms, Engagement in Care, HIV, PrEP, and STI); videos that did not address these topics were excluded. The initial list of videos included 93 CDC/Department of Health and Human Services (HHS) videos and 62 non-CDC/HHS videos from campaigns such as #AskTheHIVDoc.

A CDC study team member coordinated the initial review of videos, resulting in 155 videos that were then reviewed by six scientific review team members in three rounds of iterative review, focusing on five key areas: length of video, accuracy of scientific information, portrayal of actors or “real people”, production quality, and video genre (e.g., dramatic, documentary, street intercept/“man on the street,” animated).

In round one, 152/155 were reviewed by at least two reviewers; 119/155 videos were reviewed by at least three reviewers; 67/155 were reviewed by four or more reviewers. Three of the review team members each reviewed >75% in this round and all reviewers watched at least 25% of the videos. For each video watched, reviewers used a scoring sheet to rate its relevance to the project (Yes, Maybe, No), provide open-ended reactions to the video, and list any M-CUBED written messages that were addressed in the video. We calculated the proportion of reviewers endorsing each video by dividing the number of “yes” ratings by the number of submitted reviews for that video. The 56 highest-scoring videos were selected for further consideration.

In the second round, the goal was to identify brief videos (i.e., approximately 1 minute) that addressed the 36 core written messages that were developed as part of the study. Video length was based on recommendations from focus groups conducted at the beginning of the project and is supported by the general online marketing benchmark of up to 60 seconds to hold viewer attention (25). Based on the health messages identified by our scientific review team reviewers, we determined that the set of videos with the most comprehensive coverage was still missing six of our 36 health messages, with a duration of 26 minutes, exceeding our total targeted viewing time. In the final round, we added 15 videos with the goal of filling messaging gaps and reducing overall video duration.

Video selection and approval

Final video selection included balancing of specific criteria, described above, to ensure that they were intended for GBMSM audiences (e.g., HIV-negative men who did or did not engage in condomless anal sex; HIV-positive men). After the three video review rounds, the authors voted on a minimum of two videos per prevention domain, but also the ability to shorten the videos. Ultimately, 15 videos with a total duration of 23 minutes were selected for the formative phase—the in-depth interviews (IDIs) (see Table 1) (26-40). These publicly available videos, which were selected by the scientific review team and approved for the intervention, were previously developed by the CDC and the Kaiser Family Foundation. We received permission from each source to edit their videos for the present study.

Study design

In 2017, we conducted 26 IDIs to assess GBMSM’s reactions to the pre-selected video-based prevention messages, as well as participants’ identification of intended messages, message comprehension, and the willingness to view the video messages. The interviews were intended to inform final video message selection for inclusion in the RCT of the M-Cubed intervention.

Study population and recruitment

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). This research was reviewed and approved by the Emory University Institutional Review Board (Protocol IRB00085716) and informed consent was taken from all the participants. Participants were eligible to participate if they identified their gender and sex at birth as male (cis-gender male); were aged 18 years or older; resided in Atlanta, GA (ATL), Detroit, MI (DET), or New York City, NY (NYC); and reported anal sex with a male partner in the past six months. Each of the study sites is located in a different U.S. region and was chosen for the RCT phase based on estimates of new HIV diagnoses, including among racial and ethnic minority GBMSM who are overrepresented in the epidemic (41-46). Given that videos in the RCT were to be delivered through a mobile phone app, participants were also required to own an Android or iOS smartphone. Additionally, the funding announcement required us to reach GBMSM across HIV status/transmission risk categories. Thus, participants were recruited to represent three groups: self-reported HIV-negative men who were at lower risk of HIV acquisition, self-reported HIV negative men who were at higher risk of HIV acquisition, and self-reported men living with HIV. Men at lower risk of acquiring HIV were those who reported 100% consistent condom use in the three months prior to screening and/or current PrEP use. Men at higher risk of HIV acquisition were those who reported less than 100% consistent condom use and not currently taking PrEP. HIV serostatus was self-reported and was not validated via HIV testing or medical records.

Participants were recruited using banner advertisements placed on social media websites such as Facebook. Banner advertisements were linked to a page with information describing the study’s purpose. After reading this information, potential participants completed an online eligibility screener that included questions on age, residence, sex at birth, gender identity, sexual behavior, and HIV status. Those who screened eligible provided contact information and were invited to participate in the research.

Procedures

In each study city, IDIs were held at a central location familiar to the population (i.e., HIV Service Organization, university, or research institution). Upon arrival at the study site, participants were rescreened to confirm eligibility and risk group, and if eligible, completed the informed consent process. During the consent process, participants had an opportunity to ask questions about the study procedures and their participation.

During the IDI, participants were shown each of the 15 videos with sexual health or HIV prevention messages on a tablet. The videos were divided into groups based on the prevention domain the messages in the video targeted. Both the order of the domains and the videos within each domain were randomized so that each participant was presented with a unique domain and video message order. This was to reduce any effect that presentation order could have on perceptions of each video.

The IDI question guide was structured to capture elements of videos that prior research has shown to be important drivers of comprehension and willingness to view (24). The IDI questioning centered on overall perceptions of the videos, and the questions were based on the six dimensions of the videos: length of video, perceived accuracy of scientific information, reactions to the use of actors or “real people” portrayed, production quality, and reactions to the video genre (e.g., dramatic, documentary, street intercept/“man on the street”), and perceptions of the use of animation versus live actors. Interviewers asked for reactions to each of the videos in turn, with probes for each of the six dimensions listed above, and were asked to identify the intended HIV prevention or care messages, comprehension of those messages, and willingness to view the video. Probing also included asking about specific qualities of the video, such as length, style, language used, and how they might be made more effective.

Data analysis

Audio recordings of all 26 video IDIs were transcribed, reviewed for quality assurance, and entered into ATLAS.ti and MAXQDA for coding. Both qualitative software programs were used based on site availability, but this did not impact overall coding, data extraction, or analysis. Initially, the lead author (MD) and one of the study site coordinators (IY) applied the text-based message IDI coding scheme to the first three video IDI transcripts for the NYC site. However, the text-based and video IDIs tapped into different elements of HIV/STI prevention messaging delivery. Thus, members of the study team made the determination that a new coding scheme would be necessary for the video IDI transcripts.

Independently, the two researchers (MD, IY) systematically read the same three video IDI transcripts for the NYC site and developed preliminary codes using a process of qualitative content analysis (47). Preliminary codes and subcodes were compared and discussed to arrive at a mutually agreed upon set of codes. A third researcher (AH), also familiar with the video IDI transcripts, applied this coding scheme to the same three transcripts for further refinement. All three researchers reconciled differences and updated the coding scheme. Then, the updated coding scheme was applied to all 26 video IDI transcripts. Researchers combined some of the codes during this process. Themes were based on patterns observed in the data, particularly relationships among individual codes. Members of the research team (SW, RZ, MD, SH) finalized theme descriptions and selected a subset of participant responses to illustrate each of the themes derived from our video IDI analysis.

Results

Sample characteristics

Twenty-six participants completed interviews in this study: nine in Atlanta, nine in New York City, and eight in Detroit. Of these 26 participants, nine were living with HIV, eight were in the HIV-negative lower risk group, and nine were in the HIV-negative higher risk group. Twelve individuals (46.2%) identified as Black non-Hispanic, one (3.8%) as Black Hispanic, seven (26.9%) as white non-Hispanic, three (11.5%) as Asian non-Hispanic, and three (11.5%) as Hispanic. Participants ranged in age from 19 to 62 years with a mean of 29 years.

Coding scheme

The coding scheme included 16 codes, which were collapsed into 13 codes. Based on this revised coding scheme, the study team identified five themes described below: participant reactions, message comprehension, PrEP concerns, targeting of video messages, and prompted action. The study team selected quotes that best reflected a particular theme or subtheme.

Participant reactions

Across the 15 videos, participants had varying reactions to the video subject matter, characters, and style. The theme participant reactions captures participant feelings, thoughts, suggestions, or concerns about video content; subthemes include: positive reaction (elements of the video that the participant liked), memorable (participants’ perceived ability to remember video content after watching it), relatable (the extent to which the participant felt themselves or other viewers could relate to the characters or messages seen in the videos), empathy (participant noted an empathic response to the video, character, or situation), and negative reaction (e.g., elements of the videos that the participant did not like).

Positive reaction

Participants commented positively about video style, key messages, and character traits. Several participants remarked favorably about videos that emphasized the importance of having a good relationship with medical providers. One participant specifically discussed provider and family support related to HIV care and LGBTQ health concerns:

“This is there. So, I like that he talked about having a friend, having some support in that process. I liked that he made a point to share that, you know, in his situation where he didn’t really know either much about or what to do afterwards. That he was surrounded with support, not only from his parents but from his doctor. And that just goes to show that we’re demonstrating the parents, what people, all the nature of I guess care in some situations. Yeah. Because a lot of people are afraid of the doctors and a lot of people have apprehensions, things of that nature. But to share that, to share an experience of actually being carried, right, by your doctor is a very powerful thing. So, something we don’t see doctors like that. I think that doctors have a great responsibility to lift people up and to help them through with their present circumstances…”—Atlanta, Black Male, 20, HIV-Positive, Video 13.

Memorable

Most participants recalled specific memorable content from videos, including certain stylistic elements such as the use of humor:

“I think a lot of the content was good, like when he said that, ‘I just make [HIV testing] a routine. It’s something that I just do.”—New York, Asian Male, 24, HIV-Negative Higher Risk, Video 8.

“You know, like I think it sort of poked fun at itself, like it would—yeah, I think it was—I thought it was good. I mean, it’s—it’s hopeful—well, you see, the other problem is like I went to a school where, like, condom usage was talked about in high school and stuff. I mean, like we started with condoms in like middle school or maybe even earlier than that, like California was pretty up there with its like sex education. So, like, this didn’t tell me anything I didn’t already know, I guess, but I kind of like the humor of it.”—Atlanta, White Male, 28, HIV-Negative Lower Risk, Video 1.

Relatable

The subtheme relatable illustrates how participants connected to the characters within the videos, including their HIV status, sexual orientation, race/ethnicity, personality traits, and behaviors. For example, this individual reflected on his own relationship and HIV testing behavior:

“So, it’s just something that I can more relate to being like, okay, I understand. You know, I’m in a relationship and I get regularly tested.”—New York, White Male, 24, HIV-Negative Lower Risk, Video 8.

Some participants indicated that race was an important trait that helped make a character relatable. And, as this example shows, the participant’s ability to connect/relate to the character portrayed in the video motivated him to act:

“Again, not specific like I mean I feel like yes, the fact that it was him -- it was a person of color I feel like I could relate to him very, very much. And I feel like that definitely just encourage me to like take the actual step in that direction, yeah.”—New York, Hispanic Male, 19, HIV-Negative Higher Risk, Video 13.

Empathy

Some participants empathized with a character in the video, particularly noting the difficulty disclosing one’s HIV status or sexual orientation. One individual put it this way:

“Well, it touched my heart first off. And I like that fact that he said—you know being gay was, some people don’t look at it very well. And the fact that he finding out that he was HIV positive, it was hard to tell people about it, which I think that’s one of the hardest – I think that’s hard…coming out actually is telling someone you are HIV positive.”—Detroit, Black Male, 25, HIV-Negative Higher Risk, Video 15.

Similarly, another participant empathized with a video character and wanting to protect themselves and their partners from HIV:

“It was just a story, and you know -- just -- of -- of what happened. I mean, you -- you kind of feel compassionate or sympathetic for the guy. But you know for someone that may possibly have HIV, you know, they might want to, you know -- what they can do to try to, you know, protect themselves or protect their partner or what not. But in this video, right here, I think it did that as far as giving you solutions, instead of just, you know, an actual story.”—Atlanta, Black Male, 30, HIV-Negative Lower Risk, Video 12.

Negative reaction

Video elements disliked by some participants included the style and content, certain language used, video length, and authenticity:

“But it felt like - it really felt like a drug ad, like I was like ‘Oh! Gosh don’t start saying the symptoms and stuff” and then it started saying the symptoms and stuff which I mean it’s important to know but I don’t know, drug ads are just really weird. So...”—Atlanta, White Male, 24, HIV-Negative Higher Risk, Video 4.

“They just said PrEP. They didn’t, like, really explain what PrEP was. I don’t know how much I’m like -- just sounds like a robot is talking, but it’s not a robot. I know it’s a person, but I don’t know how much. I think I would have exited out of these commercials, because it’s kind of like those spam commercials you get. I don’t know if his voice is used a lot, but you just hit everything like ooh, this is like a spam. (Laughter) That was my takeaway from it, but other than that, it was brief. It was to the point, I like it.”—Atlanta, Black Male, 22, HIV-Negative Lower Risk, Video 6.

Message comprehension

Message comprehension represents the extent to which participants grasped the intended messages within the videos. In general, participants successfully identified the intended messages across all six messaging domains and across video types. For example, regarding HIV testing, most participants identified the importance of regular testing, with two participants from Detroit stating:

“You know you should get tested multiple times a year, make it a part of your you know regular health care regimen, all those kind of things like I did hear those messages.”—Detroit, Black Male, 37, HIV-Positive, Video 10.

“If you are a gay man having sex with other gay men, you need to not think about getting tested as a task, it just needs to be part of your daily life […] So go get tested and don’t even think about it as going to get tested. Don’t think about it as a hassle. Just think about it as part of your life.”—Detroit, Black Male, 28, HIV-Negative Lower Risk, Video 10.

For videos depicting messages within the ART and Engagement in Care domains, several participants highlighted the need for individuals to be informed about HIV and their own sexual health. Typically, this was framed as the individual (viewer) taking charge of their own health or as a necessity in case their provider was not an HIV specialist:

“That you might be unfortunate enough to have a provider who isn’t as knowledgeable about HIV as they should be. And how that’s why it’s really up to you to be as informed as you can and use all of the tools at your disposal to inform yourself to like as much as you can because you can’t always rely on your doctor for all of your information.”—New York, Hispanic Male, 19, HIV-Negative Higher Risk, Video 13.

Further, participants were able to describe the implications of having an undetectable viral load, described in one of the videos, as shown in this example:

“That if you were undetectable the chances of someone contracting the virus from you is practically zero. And that there are other ways you know if someone is still concerned that the other person is undetectable that you can still use condoms for use perhaps to reduce the risk even further”—Detroit, White Male, 62, Detroit, HIV-Positive, Video 12.

Finally, participants were able to infer messages not explicitly stated within the videos. For example, one individual from Atlanta discussed the relationship between self-worth and HIV testing:

“I guess that the one that struck me the most was like “You’re worth it.” I guess like the message is it’s important to care enough about yourself that you would get tested, rather than not caring, and just sort of like letting the disease progress.”—Atlanta, White Male, 25, HIV-Positive, Video 9.

PrEP concerns

Several PrEP concerns were identified based on participants’ discussions about the five PrEP videos they watched during their IDIs. Two sub-themes emerged: PrEP misconceptions and PrEP skepticism. Participant misconceptions about PrEP included personal or public misunderstandings, and skepticism referred to any disbelief of PrEP as a biomedical prevention strategy for HIV.

PrEP misconceptions

From watching the PrEP domain videos, some participants expressed concern for others’ limited understanding of PrEP. For example, this individual remarked that PrEP should be taken anywhere from 4 to 7 times per day:

“So, this is my question. So, I know he said this is going to take three to four days, and of course, being medical professionals they shouldn’t recommend that someone takes it less than the amount they’re supposed to take. But I thought that research has shown that taking it four times a day gives you the same level of protection as taking it seven times a day, even though they recommend you should take it seven times a day. So, is that like false, or is that true”?—Atlanta, Black Male, 22, HIV-Negative Lower Risk, Video 2.

In another example, this participant, who was living with HIV, remarked that a particular video should clearly explain for viewers that PrEP only prevents HIV and does not protect against other STIs:

“That was good, it was very informative. I liked that. My only concern that I think should have probably been addressed is that a lot of people think PrEP stops everything. They think it stops more than just HIV. And I think that should be addressed as like “This will not prevent other STDs, or STIs.” But, other than that, I think it was fine.”—Atlanta, White Male, 25, HIV-Positive, Video 5.

Similarly, another participant discussed the lack of condom use among his friends who are taking PrEP as a misconception of the protection afforded by PrEP:

“Because I mean I just worry about my friends because, you know, they’re not, you know, they’re not using condoms. And they feel like it’s okay because they’re on PrEP.” New York, White Male, 24, HIV-Negative Lower Risk, Video 4.

In the passage below, a participant ties PrEP use with sexual fluidity, suggesting that anyone who has sex should take PrEP until there is no more HIV. This interpretation is divergent from how providers consider a patient’s sexual behavior in their recommendation of PrEP:

“It’s scary though how it was like you might not have to take it for the rest of your life because sexuality is fluid. So that’s like saying you know I’m settling down with this one partner, he’s negative, I’m negative we can go raw! And then so I don’t have to take the PrEP and then I have to take it later but I don’t know if I want to keep. If sexuality is fluid then it would be best to just stay on PrEP. Anybody having sex should be on PrEP until we’ve eradicated this disease.”—Detroit, Black Male, 31, HIV-Positive, Video 3.

PrEP skepticism

Participants discussed their skepticism toward PrEP as a concern about the medication’s side-effects, novelty of the drug, or appropriateness of PrEP for themselves. As illustrated below, several participants discussed not knowing if PrEP is appropriate for them based on the medication’s effectiveness and their own sexual behaviors:

“But it was also like, ‘This data doesn’t sound great. It’s 92% effective and it gives you these side effects were the two like pieces of data that stood out to me. So that made me less likely, probably, to want to go get it, especially if I was just now hearing about it.”—New York, Asian Male, 24, HIV-Negative Higher Risk, Video 4.

“Maybe they could have said it like that though like if your behavior changes you might not need it. But specifically if you change your behavior. You might not need PrEP. Then that’d be a whole another video about what behavior you need to change”—Detroit, Black Male, 31, HIV-Positive, Video 3.

Targeting of video messages

The theme targeting of video messages comprised three subthemes: cultural awareness, diversity, and credibility. Cultural awareness refers to the participants’ perceptions of how well the videos apply to individuals from different cultures or belief systems, such as various ethnic, racial, geographic, or social groups. Diversity refers to the extent to which the videos included a variety of characters from different backgrounds or races. And credibility describes the extent to which information or characters presented in the video are believable or trusted by the participant. Participant remarks within this theme were largely related to the messenger (e.g., dramatic character, healthcare provider).

Cultural awareness

Participants had mixed reactions around portrayals of cultural awareness. Some described how videos successfully showed the interplay between race and sexuality, with one stating:

“The fact that in our communities especially when it’s like a people of color circle like homophobia is so rampant and like you don’t really have anyone to talk about your sexuality let alone about the fact that you contracted something. And I feel that really hit closer home for me because I feel like I was raised in a similar environment where I didn’t really feel even now like I don’t feel comfortable with my doctor.”—New York, Hispanic, 19, HIV-Negative Lower Risk, Video 14.

Another participant commented on how the tone of the language used in a video must be considered to ensure cultural awareness:

“To the general public, like it might seem slightly aloof, very ivory tower-ish. Like there’s certain terms, like the way he describes things that like outside of academia people might not really be familiar with.”—Atlanta, White Male, 25, HIV-Positive, Video 2.

Diversity

With this subtheme, most participants commented on the importance of racial diversity in the videos selected for the intervention, particularly the inclusion of racial and ethnic minorities. One participant commented on the characters and their diversity, but noted a lack of women—though he does recognize that women may not be an intended audience.

“I think one thing that I begin to focus on in this video was the inclusivity. I was glad when I had to see a variety of faces or people who – for whom PrEP might be a great fit for. However, they – I didn’t feel that women were included in this. I know that the target may very well be, you know, men or what have you.” Atlanta, Black Male, 20, HIV-Positive, Video 6.

Some participants, however, commented that characters who are diverse in ways not related to race or ethnicity are also important to feature. For example, this individual from Detroit pointed out a certain degree of homogeneity within racial/ethnic portrayals of characters. His response is indicative of the importance of not showing all people in the same manner:

“While I feel like I’ve seen a lot of different types of people you know you seen someone was a little bit bigger, someone was smaller, seen one black person, a Latino person like, you’ve seen a lot of different I think it’s being diverse in only in one kind of way, you’re showing cultural diversity. But I don’t think it’s showing like the -- I think the black guy had like this masculine kind of tone about yourself and then everyone else’s had of a more like a feminine tone and I think that starts to send a message of everybody who’s gay is this.”—Detroit, Black Male, 37, HIV-Positive, Video 7.

Credibility

Participants often commented on the importance of having a medical professional as the messenger to add credibility to a video’s message(s). The portrayal of a physician as messenger facilitated trust by the viewer, which was fairly consistent across the sample even if a participant was unsure whether the medical professional portrayed in the video was an actor or truly a physician. These two quotes illustrate this point:

“Probably they going to say was he a doctor, okay we’re going to listen to what he’s saying, and that’s a good thing because you want a doctor to tell you exactly go get checked out.”—Detroit, Black Male, 47, HIV-Negative Higher Risk, Video 10.

“Just, I feel like the doctors do a good job of, I don’t know if they’re doctors or actors, but they just do a good job of talking about the drug and not making it seem like something that’s bad or weird or different.”—New York, Hispanic Male, 19, HIV-Negative Lower Risk, Video 3.

Of particular importance to this study, credibility of a health care provider as the messenger may be questioned by a viewer when the information presented is perceived to conflict with that of their own provider. For example:

“Yeah, because like even now when I go to my general practitioner whatever, like I’ve had two and one of them has been like, ‘You should be on PrEP because of X, Y and Z’ and then others are kind of like, you know. PrEP might be good but based on what you tell me, it doesn’t sound like you really need to be in it, you know, like – so it’s hard when you’re already getting competing messages from your own doctors to then be listening to this and then say, okay. So I have a relationship and a rapport with my doctors. I am more inclined to believe them than I am him but if he had a little bit more credibility attached to him, then”—Atlanta, White Male, 28, HIV-Negative Lower Risk, Video 3.

Prompted action

The final theme, prompted action, considers potential action or behavior brought about by a particular video or video message as indicated by participants. More specifically, individuals discussed their own potential actions as well as those that others may take after watching the video. Actions most frequently mentioned included initiating PrEP, receiving and recommending STI and HIV testing, conducting further research about a topic, finding a new healthcare provider, and sharing the video with friends.

Several participants commented on their desire to find a medical provider that they feel comfortable disclosing their sexual orientation to and talking with about their healthcare needs. After watching a character describe the relationship with his doctor, one participant indicated that he may consider taking steps to find a new medical provider:

“I liked his story. It’s nice to hear about a doctor who would want you to be such a good informed self-advocate. It makes me feel like I wish my doctor was like that. And it makes me want to ask around and see if I know anyone who does have a doctor like that to see if I can get a different provider.”—New York, White Male, 24, HIV-Positive, Video 13.

Other participants commented on wanting to receive comprehensive STI testing. For example, this individual discussed wanting a provider to swab multiple anatomical sites as part of STI testing:

“Well, I will be honest with the doctor, and I will make sure that like he said, I was going to get tested for not only HIV but all STIs and getting rectal and penis swabs. Penis swabs, I think. Not usually penis swabs now, but you can get swabbed too.”—Atlanta, Black Male, 22, HIV-Negative Lower Risk, Video 11.

The purpose of the IDI process was to determine acceptability, message comprehension, and potential audience reach for 15 videos to be used as part of the M-Cubed intervention app. Based on participant feedback during the IDIs, the research team made a final selection of 12 videos. The three videos excluded from the final selection were “What is PrEP? (Video 2)”, “Is PrEP Right for You? (Video 6)”, and “Taking Care of Me - Social Support (Video 15).” The video “What is PrEP?” (Video 2) was excluded because of inconsistent PrEP information. Specifically, at the time interviews were conducted, the video’s messaging about PrEP dosing schedule (e.g., PrEP on demand) was inconsistent with information provided in the other PrEP videos and was not FDA-approved for use in this manner. “Is PrEP Right for you?” (Video 6) and “Taking Care of Me – Social Support” (Video 15) were excluded because they received negative feedback from participants. Negative impressions from these two videos included that they were overdramatized or perceived as a commercial, detracting from the seriousness of the health messaging.

Discussion

The current report examined a systematic process of video-based message selection for an HIV/STI prevention mobile app and presented findings from a formative qualitative study to better understand video acceptability and message comprehension among a sample of HIV-negative and HIV-positive GBMSM. Findings from this analysis may serve as a guide for researchers who plan to use similar methods across research disciplines.

Participant reactions to the video messages were both positive and negative, but also raised awareness about the importance of having intervention content that is memorable, relatable, and has the ability to affect audience emotions. It is evident that there are a number of video qualities and characteristics that are essential for delivering effective HIV/STI prevention messaging. As our findings suggest, video content that is memorable, features relatable characters and storylines, or evokes an emotional response is important to engaging intended audiences and should be considered when selecting video messages (7,10,20). Indeed, messaging content that is memorable or provocative has been associated with behavior change (10). Moreover, some participants in this study indicated that race was an important factor when reflecting on the videos presented. Thus, the intended audiences for video-based interventions should be racially or culturally represented and realistically portrayed with relatable characters (10).

Overall, our formative testing of the video messages suggests comprehension of content in the six domains. More specifically, and critical to the RCT phase of this research, participants did identify many of the intended prevention and care messages offered in the set of videos. For example, some participants acknowledged the importance of routine HIV and STI testing as demonstrated in Videos 9, 10, and 14; while others paid attention to information about viral load suppression and the risk of HIV transmission as shown in Video 12.

PrEP misconceptions and PrEP skepticism were two sub-themes that emerged under PrEP concerns. Interestingly, three of the quotes were from men living with HIV and suggests the potential utility of showing PrEP informational messaging to mixed serostatus audiences. For example, one of the quotes from an HIV-positive participant identified an important gap in a video message pertaining to the protection that PrEP offers against HIV, but omits the detail that PrEP does not protect against other STIs. Additionally, another man noted that there should have been messaging about specific circumstances where a person would not need to use PrEP (e.g., changing from higher risk to lower risk sexual behavior). These findings suggest that men living with HIV may have an informed perspective that could be shared with other HIV-positive and HIV-negative partners and peers, just as HIV-negative men may benefit from being informed about HIV viral load and ART adherence. Furthermore, the formative findings support the need to provide accurate and comprehensive prevention messaging across our intended populations.

When selecting videos for intended audiences, results from this formative work suggest the importance of representation (e.g., racial and ethnic diversity). We recognize that diversity should also encompass geography, age, and income level. Our work suggests the importance of accessible messaging (i.e., culturally aware, but not “ivory tower-ish” language) and assuring audiences that the messenger is a credible source. For the latter, our participants were not always sure if the medical professional portrayed in the video was an actual doctor/physician or an actor. This is critical information to provide to intended audiences, particularly since the participants in this study would accept medical professionals as a messenger.

The theme, prompted action, is critical to the RCT phase of this study because it suggests intent by the participant to engage in some form of behavior change. For example, at least one participant indicated a desire to find a new healthcare provider after watching one of the videos from the Engagement in Care domain. This is consistent with prior eHealth interventions for GBMSM that included video messaging and formative work, resulting in actual or intended behavior change (10,20,48). Prompted action can be measured by paradata (i.e., process data) collected through an app being tested in an RCT. If a participant in the prior example were enrolled in an mHealth intervention that offered links to healthcare providers in their area, researchers could better understand that participant’s intentions to change their behavior based on actions taken within the app. Thus, by delivering relevant prevention messaging to diverse audiences about PrEP, HIV and STI testing, ART, and other health resources, interventions can test associations between intentions to change (e.g., user paradata for accessing app-based provider locater services) and actual behavior change (e.g., follow-up assessment of whether a user found a provider, made an appointment, and/or attended an appointment).

Limitations

The goal of this formative study was to compile a set of publicly available videos, match videos to HIV/STI prevention domains, and then confirm the video selection with a sample of the target population for the M-Cubed RCT. Due to budgetary and time restrictions, we opted to utilize existing videos. Thus, we were not evaluating video development, but eliciting feedback about overall message comprehension and acceptability. Healthcare and non-profit organizations may not have funds to develop new prevention materials and there is a wealth of available video messaging online.

The advantages of utilizing these particular videos included that they were publicly accessible, were all vetted and endorsed for scientific campaigns, and we were able to edit them for the purpose of this study. In this research, which represented a cooperative effort co-led by academic and government researchers, we prioritized inclusion of video materials already reviewed by CDC to streamline the review timeline and to expedite the start of the trial. We recognize that by utilizing federally approved videos, we limited the scope of health topics covered and there was less use of realistic language and visual elements (e.g., limited or no swearing, no shirtless males or simulated sex) (23). We acknowledge this as a limitation of our approach and also recognize that this description of our approach to video selection may be useful to other researchers screening videos for research or programs supported by non-federal funding sources.

Some of the videos considered were produced by the federal funding agency responsible for approving the final selection, and therefore were less likely to cause delays. The preference for federally sourced videos might represent or be perceived as a conflict of interest given the federal funding source and federal investigators for this project. Future studies may benefit from considering video accessibility (e.g., publicly available) and editability prior to the vetting process. Also, the videos were only available in English, which limits their accessibility to some potential participants. Further, it is important to acknowledge that, as with other public health issues, historically, HIV prevention has existed within the context of systemic racial bias. However, sound strides were made during the video selection process to include established videos from formative work with target audiences.

We sought to include videos that addressed the range of health outcomes targeted by the app (e.g., HIV, STIs), though most of the video content available for inclusion was HIV-focused. Also, some of the videos reviewed by participants addressed overlapping topics; thus, it is possible that participants perceived different guidance on similar topics (e.g., quotes about Videos 2 and 12). Further, we acknowledge that participants in this formative phase of the study did not have access to the app or its features. However, it is important to note that all participants in the planned RCT phase of the study would have access to other resources within the app and an opportunity to communicate with study staff about any questions that might have arisen.

Our data were subject to selection bias. We included only men from Atlanta, New York City, and Detroit; this choice was made because those cities have been heavily impacted by the HIV epidemic and were the three locations where the RCT was to be conducted. Therefore, we acknowledge that our results might not be generalizable to GBMSM in other cities. Further, we recruited men based on their status as GBMSM, but the video content focused only on sexual risks. We acknowledge that GBMSM also have other risks for infection, including related to injection drug use (IDU). Future studies should consider including videos that address other concurrent risks, including IDU, given that risks from male-male sex and IDU often impact the same men (49).

We could not target videos by age group—most of the videos portrayed men in their 20s and 30s, except for the #AskTheHIVDoc videos, which portrayed slightly older men. Targeted video messaging by stage of life may enhance the efficacy of intervention messaging. Our analytic strategy focused on coding and themes across the full sample of participant IDIs. An alternative strategy could have been to focus on coding responses by each video. However, we chose not to code by video because we did not develop and conduct initial testing of the videos. Finally, we did not systematically focus on video style, mood, tone, actor portrayal (and perceptions about the actor), or character development/storyline. Thus, we are not able to examine these elements in relation to message comprehension. Future video-based intervention efforts should consider qualities of the video production in determining whether the presented messages are understood, as intended, by the viewing audience.

Conclusions

We described a process for selecting and evaluating videos for inclusion in an app-based intervention to promote HIV/STI prevention and care for GBMSM. By seeking input from potential users of the intervention app early in the decision-making process, we helped to increase the potential for engagement with the intervention content during the intervention trial. We recommend systematic assessment of participant engagement with videos through analysis of paradata and assessment of participant ratings of videos using data from the M-cubed trial. Based on our results, it is important to promote the development of videos with broad age and gender diversity for use in interventions such as ours, and in other health promotion settings.

Acknowledgments

We thank the M-cubed Study staff and the CDC and Kraft labs. We are grateful to the M-cubed study participants and community advisory board members. The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the US Centers for Disease Control and Prevention.

Funding: This work was supported by the Centers for Disease Control and Prevention (Grant Number U01PS004977) and the National Institutes of Health (Grant Number P30AI050409).

Footnote

Data Sharing Statement: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-21-53/dss

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-21-53/coif). PSS reports that his funding from CDC and NIH are payments made to his institution for other research projects, and they are not related to the present work. ISY has been a full-time employee of Mattel Global Consumer Insights since 2021, but received no support related to this publication. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). This research was reviewed and approved by the Emory University Institutional Review Board (Protocol IRB00085716) and informed consent was taken from all the participants.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Centers for Disease Control and Prevention. HIV Surveillance Report, 2018. Available online: https://www.cdc.gov/hiv/library/reports/hiv-surveillance.html [accessed 2021-02-25]
2. Marcolino MS, Oliveira JAQ, D'Agostino M, et al. The Impact of mHealth Interventions: Systematic Review of Systematic Reviews. JMIR Mhealth Uhealth 2018;6:e23. [Crossref] [PubMed]
3. Sullivan PS, Driggers R, Stekler JD, et al. Usability and Acceptability of a Mobile Comprehensive HIV Prevention App for Men Who Have Sex With Men: A Pilot Study. JMIR Mhealth Uhealth 2017;5:e26. [Crossref] [PubMed]
4. Fernandez MI, Harper GW, Hightow-Weidman LB, et al. Research Priorities to End the Adolescent HIV Epidemic in the United States: Viewpoint. JMIR Res Protoc 2021;10:e22279. [Crossref] [PubMed]
5. Holloway IW, Rice E, Gibbs J, et al. Acceptability of smartphone application-based HIV prevention among young men who have sex with men. AIDS Behav 2014;18:285-96. [Crossref] [PubMed]
6. Warner L, Klausner JD, Rietmeijer CA, et al. Effect of a brief video intervention on incident infection among patients attending sexually transmitted disease clinics. PLoS Med 2008;5:e135. [Crossref] [PubMed]
7. Hirshfield S, Chiasson MA, Joseph H, et al. An online randomized controlled trial evaluating HIV prevention digital media interventions for men who have sex with men. PLoS One 2012;7:e46252. [Crossref] [PubMed]
8. Pew Research Center. Mobile phone ownership over time: 2019 Pew Research Center. Available online: https://www.pewresearch.org/internet/fact-sheet/mobile [accessed 2021-03-20]
9. Chiasson MA, Hirshfield S, Rietmeijer C. HIV prevention and care in the digital age. J Acquir Immune Defic Syndr 2010;55:S94-7. [Crossref] [PubMed]
10. Chiasson MA, Shaw FS, Humberstone M, et al. Increased HIV disclosure three months after an online video intervention for men who have sex with men (MSM). AIDS Care 2009;21:1081-9. [Crossref] [PubMed]
11. Blas MM, Alva IE, Carcamo CP, et al. Effect of an online video-based intervention to increase HIV testing in men who have sex with men in Peru. PLoS One 2010;5:e10448. [Crossref] [PubMed]
12. Rosser BR, Hatfield LA, Miner MH, et al. Effects of a behavioral intervention to reduce serodiscordant unsafe sex among HIV positive men who have sex with men: the Positive Connections randomized controlled trial study. J Behav Med 2010;33:147-58. [Crossref] [PubMed]
13. Cohen DA, Wu SY, Farley TA. Comparing the cost-effectiveness of HIV prevention interventions. J Acquir Immune Defic Syndr 2004;37:1404-14. [Crossref] [PubMed]
14. Mustanski B, Parsons JT, Sullivan PS, et al. Biomedical and Behavioral Outcomes of Keep It Up!: An eHealth HIV Prevention Program RCT. Am J Prev Med 2018;55:151-8. [Crossref] [PubMed]
15. Collins A. Design issues for learning environments. In: Vosniadou S, De Corte E, Glaser R, Mandl H, editors. International perspectives on the design of technology-supported learning environments. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.; 1996:347-61.
16. Bransford J, Sherwood R, Hasselbring T, Kinzer CK, Williams SM. Anchored instruction: Why we need it and how technology can help. In: Nix D, Spiro R, editors. Cognition, education, and multimedia: Exploring ideas in high technology. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.; 1990:115-141.
17. Rivera AV, DeCuir J, Crawford ND, et al. Factors associated with HIV stigma and the impact of a nonrandomized multi-component video aimed at reducing HIV stigma among a high-risk population in New York City. AIDS Care 2015;27:772-6. [Crossref] [PubMed]
18. Tuong W, Larsen ER, Armstrong AW. Videos to influence: a systematic review of effectiveness of video-based education in modifying health behaviors. J Behav Med 2014;37:218-33. [Crossref] [PubMed]
19. Calderon Y, Cowan E, Leu CS, et al. A human immunodeficiency virus posttest video to increase condom use among adolescent emergency department patients. J Adolesc Health 2013;53:79-84. [Crossref] [PubMed]
20. Hirshfield S, Downing MJ Jr, Chiasson MA, et al. Evaluation of Sex Positive! A Video eHealth Intervention for Men Living with HIV. AIDS Behav 2019;23:3103-18. [Crossref] [PubMed]
21. Grov C, Breslow AS, Newcomb ME, et al. Gay and Bisexual Men’s Use of the Internet: Research from the 1990s through 2013. J Sex Res 2014;51:390-409. [Crossref] [PubMed]
22. Maloney KM, Bratcher A, Wilkerson R, et al. Electronic and other new media technology interventions for HIV care and prevention: a systematic review. J Int AIDS Soc 2020;23:e25439. [Crossref] [PubMed]
23. Muessig KE, Nekkanti M, Bauermeister J, et al. A systematic review of recent smartphone, Internet and Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep 2015;12:173-90. [Crossref] [PubMed]
24. Sullivan PS, Zahn RJ, Wiatrek S, et al. HIV Prevention Via Mobile Messaging for Men Who Have Sex With Men (M-Cubed): Protocol for a Randomized Controlled Trial. JMIR Res Protoc 2019;8:e16439. [Crossref] [PubMed]
25. Cheng X, Liu J, Dale C. Understanding the characteristics of internet short video sharing: A YouTube-based measurement study. IEEE Trans Multimedia 2013;15:1184-94. [Crossref]
26. AIDS Healthcare Foundation. Public Service Announcement: How to Use A Condom. 2009. [Video 1]
27. Kaiser Family Foundation. #AskTheHIVDoc: What is PrEP? 2015. Greaterthan.org. [Video 2]
28. Kaiser Family Foundation. #AskTheHIVDoc: How Effective is PrEP? 2015. Greaterthan.org. [Video 3]
29. Centers for Disease Control and Prevention. #Start talking, Stop HIV: Excerpt, “What is PrEP?” PrEP - An HIV Prevention Option. FHI360.org [Video 4]
30. Centers for Disease Control and Prevention. #Start talking, Stop HIV: Excerpt, “How to get PrEP” PrEP - An HIV Prevention Option. FHI360.org [Video 5]
31. Centers for Disease Control and Prevention. #Start talking, Stop HIV: Excerpt, “PrEP, Right for You?” PrEP - An HIV Prevention Option. FHI360.org [Video 6]
32. Centers for Disease Control and Prevention. #DoingIt: Alfredo Flores. 2015. [Video 7]
33. Centers for Disease Control and Prevention. #Doingit: Andrew. 2015. [Video 8]
34. Centers for Disease Control and Prevention. #Doingit: Young Black Gay Men’s Leadership Initiative (YGBLI). 2017. [Video 9]
35. Kaiser Family Foundation. #AskTheHIVDoc: How often should I get tested? 2015. Greaterthan.org. [Video 10]
36. Kaiser Family Foundation. #AskTheHIVDoc: Swab my butt! 2015. Greaterthan.org [Video 11]
37. Kaiser Family Foundation. #AskTheHIVDoc: Can I Get HIV if My Partner is Undetectable? 2015. Greaterthan.org. [Video 12]
38. U.S. Department of Health & Human Services. #PositiveSpin: Patrick - Connect to Care: This is your life. You Need to be Educated about HIV. 2016. HIV.gov. [Video 13]
39. Centers for Disease Control and Prevention. #Taking Care of Me: Doctor Communication. 2016. [Video 14]
40. Centers for Disease Control and Prevention. #Taking Care of Me: Social Support. 2016. [Video 15]
41. Centers for Disease Control and Prevention. Diagnosed HIV infection among adults and adolescents in metropolitan statistical areas—United States and Puerto Rico, 2011. 2013.
42. Centers for Disease Control and Prevention. HIV Surveillance Report, 2013; vol. 15. Available online: http://www.cdc.gov/hiv/topics/surveillance/resources/reports/. Published February 2015.
43. New York City HIV Epidemiology and Field Services Program. HIV Surveillance Annual Report, 2013. 2014. Available online: http://www.nyc.gov/html/doh/downloads/pdf/dires/2013-hiv-surveillanceannual-report.pdf.
44. Michigan Department of Community Health. Annual HIV Surveillance Report, Michigan. 2014. Available online: http://www.michigan.gov/documents/mdch/January_2014_ALL_446611_7.pdf
45. Sullivan PS, Peterson J, Rosenberg ES, et al. Understanding racial HIV/STI disparities in black and white men who have sex with men: a multilevel approach. PLoS One 2014;9:e90514. [Crossref] [PubMed]
46. Sullivan PS, Rosenberg ES, Sanchez TH, et al. Explaining racial disparities in HIV incidence in black and white men who have sex with men in Atlanta, GA: a prospective observational cohort study. Ann Epidemiol 2015;25:445-54. [Crossref] [PubMed]
47. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005;15:1277-88. [Crossref] [PubMed]
48. Klein CH, Kuhn T, Huxley D, et al. Preliminary Findings of a Technology-Delivered Sexual Health Promotion Program for Black Men Who Have Sex With Men: Quasi-Experimental Outcome Study. JMIR Public Health Surveill 2017;3:e78. [Crossref] [PubMed]
49. Nerlander LMC, Hoots BE, Bradley H, et al. HIV infection among MSM who inject methamphetamine in 8 US cities. Drug Alcohol Depend 2018;190:216-23. [Crossref] [PubMed]