Online recruitment of youth for mHealth studies

Introduction

Young men who have sex with men (YMSM) between the ages of 13 and 24 are among the groups most vulnerable to HIV acquisition in the United States (US) (1). A growing proportion of youth access the Internet multiple times a day and an overwhelming majority use social networking sites and applications (apps), such as Facebook, Instagram and Snapchat (2,3). Mobile health (mHealth) strategies have emerged as promising approaches for HIV prevention interventions targeted towards YMSM, in large part due to the nearly universal uptake of these technologies among US youth (2). To support the implementation of these mHealth interventions, researchers have sought to leverage social media platforms to facilitate the recruitment of YMSM into research trials, supplementing and at times, replacing print media and venue-based sampling methods (4-8). The use of social media platforms for recruitment presents an opportunity to reach YMSM at risk of HIV infection who may not be otherwise reached by in-person recruitment efforts, such as youth not connected to local LGBTQ community events or sexual health clinics (9-14). Thus, researchers can leverage advances in technology and digital media for novel recruitment methods as well as interventions among youth at high risk for HIV infection. Using data and case studies from several on-going HIV mHealth trials focused on enrolling YMSM across the US, this paper highlights lessons learned, best practices and on-going recruitment challenges evidenced in the iTech network of the NIH Adolescent Trials Network for HIV Interventions (15).

Paid digital advertising offers access to audiences not otherwise available through organic reach, i.e., users that have chosen to “like” or follow a given social media channel. Google and Facebook, in particular, have pioneered online tools that enable advertisers to directly purchase banner ads (16), i.e., image or video-based advertisements displayed on a website or app, with costs that are accessible to health research budgets. Furthermore, the vast amount of information these companies collect on their users’ interests and online behavior allow for refined targeting of ads to individuals who are most likely to engage with them (16,17). Most advertising platforms operate on a “pay-per-click” model and therefore cost-effectiveness depends on maximizing clicks from those who match study eligibility criteria while minimizing clicks from those who do not. Thus, regardless of the study population, effective recruitment in online spaces using paid advertising necessitates targeting advertisements so they are seen by as many potentially eligible participants as possible, while minimizing delivery to ineligible users.

The ultimate goal of most online research recruitment campaigns is to direct users to an online eligibility screener, then enroll participants directly from the screener or by banking eligible contacts for follow-up and scheduling by study staff. The relative success of ad campaigns can be assessed across platforms and ads by comparing the total advertising costs divided by the number of enrolled participants or banked contacts. Both the numerator and denominator of this equation can be influenced by many factors. For example, costs set by advertising platforms are not fixed; they are set by a complex algorithm that considers audience size, competition from other advertisers for the same audience, and budget (18). The proportion of participants who screen eligible is determined by the stringency of the eligibility criteria and overlap between ad audience and eligibility criteria. Therefore, to fully understand and optimize the dynamics of an online recruitment campaign, one must consider the totality of the process from the start of an ad being shown to the point at which an eligible participant provides contact information or enrolls in a given study (Figure 1). This necessitates the integration of metrics from online advertiser platforms and eligibility screeners so they can be presented within a consistent reporting framework.

Figure 1 Flow diagram of cascade between ad launch and study enrollment, with data source (ad platform or eligibility screener) for various metrics indicated.

At present, little is known about the comparative performance of different social media platforms, especially emerging ones like Snapchat (19), and dating/sex-seeking apps for MSM with respect to recruitment of high-risk YMSM for research. Using performance metrics from iTech studies, which concurrently recruits participants for various mHealth intervention trials, with slightly varying eligibility criteria across many US cities, provides an ideal opportunity to evaluate recruitment campaigns across a variety of online platforms and geographic markets. Herein, we report online recruitment and screening metrics for four iTech randomized controlled trials (RCTs), recruiting HIV-negative or unknown status YMSM in 10 US cities.

Methods

Studies

Recruitment was conducted for four iTech studies: ATN 139: Get Connected (20), ATN 140: LYNX (21), ATN 141 MyChoices (22), and ATN 142: P3 (Prepared, Protected, empowered) (23). All studies involved RCTs of mHealth interventions to reduce HIV transmission and included the following eligibility criteria: self-report being HIV-negative (and/or HIV status unknown for some studies) at screening, ability to speak and read English, access to the internet or a smartphone, and residence in one of the study’s recruitment sites. Individual study eligibility criteria and other characteristics are presented in Table 1.

Table 1 Summary of eligibility criteria and study characteristics for iTech studies employing online recruitment
Full table

Ad campaign setup

The data presented are from paid online recruitment campaigns and do not include organic reach through posting by site staff on various social media channels or in-person recruitment efforts. A variety of social media platforms with self-serve advertising tools were used to create advertising campaigns for each study. Details for each individual ad platform are provided below, and all allow advertisers to create ads to drive traffic to an eligibility screener. All advertising content was approved by the University of North Carolina at Chapel Hill Institutional Review Board. Screenshots of selected ads are included in Figure 2.

Figure 2 Selected examples of online advertisements. (A) Facebook ad for MyChoices study; (B) Instagram ad for MyChoices study; (C) Stills from Snapchat video ad for Get Connected study; (D) Grindr ad for LYNX study; (E) Twitter ad for P3 study.

Facebook/Instagram

Facebook is a social networking service with over 2 billion users worldwide (24). Facebook acquired the image-sharing platform Instagram in 2012, and advertisers can use its self-serve advertising management tool, Facebook Ad Manager, to create, target, and monitor advertising on both Facebook and Instagram (25). Facebook Ad Manager uses a nested structure of ads within ad sets within a campaign for a given recruitment goal (i.e., study).

Ad sets define the daily budget and ad audience. The proprietary Facebook ad algorithm uses advertiser defined settings to determine which users are shown the ads in that ad set (17). For each ad set, audience settings for gender, age, and location were set to match each study’s eligibility criteria. Though one study (P3) was also open to transgender women, the ad campaigns presented herein were all targeted at those who indicated their gender as “male” on their social media profile. We further targeted ad sets using interest keywords so that they would be more likely to be viewed by YMSM. These keywords are generated from users’ pages and post likes, browsing history, and previous ad clicks. Keywords used to define a YMSM audience included celebrities and influencers, LGBTQ media outlets, LGBTQ advocacy groups, and pop culture popular in the LGBTQ community. Lists of interest keywords were generated using previous experience in Facebook advertising, suggestions from recruitment site staff and iTech youth advisory boards. Audience targeting settings were changed periodically to refresh the audience seeing the ads and prevent audience fatigue.

Ad sets created in the Facebook Ads Manager can be shown in multiple placements on Facebook properties, e.g., News Feed, right-hand column, Instagram feed and stories. For most ad sets presented here, the default setting of automatic placements was used, meaning that the Facebook proprietary algorithm allocated the distribution of ad placements that would provide the best performance. Thus, ads were placed on Facebook and Instagram concurrently. For some ad sets, we selected only the Instagram placements to test ad performance when served exclusively on this platform.

Within each ad set, one or more ads with different assets (i.e., images and text) were created. Images used in ads included stock photos featuring young men of various races/ethnicities, either alone or in male-male couples. Stock photos were either purchased from online stock photo vendors (e.g., Shutterstock) or accessed through Facebook’s free stock photo catalog integrated within the Facebook Ads Manager. All ads created in Facebook Ads Manager included a headline (e.g., “Ever think of testing at home?”) and body text (e.g., “Fenway Health is looking for young men to help test out a mobile app to support sexual health. Earn up to $125 and help the community!”). Ad copy was written to give a very brief description of the study, identify the organization conducting study recruitment, mention the study incentive, and provide a call to action for potential participants seeking to learn more.

Snapchat

Snapchat is a multimedia messaging and social networking mobile app with approximately 210 million daily active users (26). Though its user base is smaller than Facebook’s, Snapchat is more popular among US teens (2). Advertisers can place ads within the Snapchat “Discover” feature, which shows ad-supported short-form content from various creators and brands. Advertising on Snapchat follows a nested structure similar to Facebook: ads within ad sets within campaigns.

Options for targeting audiences are more limited on Snapchat’s ad platform as compared to Facebook’s. In addition to age, gender, and location, ads can be targeted to platform predefined audiences, such as “Fashion and Style Gurus” and “Men’s Lifestyle.” The algorithm used to create these predefined audiences is not accessible to advertisers. Snapchat ads were created using Snap Publisher and were in the format of short videos up to 10 seconds long (27). These videos rotated through various stock photo images and superimposed text. Ad images and copy were similar to those used in Facebook ads.

Grindr

Grindr is a geosocial networking and online dating mobile app aimed towards gay, bisexual, trans, and queer people, primarily cisgender men. Their self-serve advertising platform allows advertisers to purchase banner ads in different sizes for display in the mobile app (28). Audience targeting options in the platform are limited to location, i.e., ZIP code, city, state, or country. Full-screen 320×480 pixel banner ads featuring stock photos and copy were used for study campaigns.

Twitter

Twitter is a microblogging and social networking service that displays banner ads in users’ feeds on desktop and mobile versions of the site, as well as mobile app. The self-serve advertiser platform has a nested structure similar to that of Facebook and Snapchat (29).

Ad audiences were created by targeting 18- to 24-year-old men living in cities with study sites, along with interest keywords similar to those used on Facebook. Only the P3 study advertised using Twitter.

Eligibility screening

Upon clicking an ad, potential participants were taken to a study-specific eligibility screener, hosted on SurveyGizmo, a HIPAA-compliant survey data collection instrument. The first page (or “landing page”) briefly described the study and included a short form to obtain participants consent for screening. Users who affirmed consent were then taken to a brief eligibility screener. Those who screened eligible for an individual study were asked to provide contact information (name, email address, telephone number) for follow-up and scheduling of enrollment visits by respective study site staff.

For each study, recruitment source by advertising platform was tracked in screener metadata by using unique links to eligibility screeners for each platform. Additionally, for some ad campaigns, screener links also identified the specific ad that had been clicked in order to arrive at the eligibility screener. In this way, recruitment metrics from the advertising platform and the survey platform could be linked in aggregate. No identifiable information was transferred from an individual’s social media profile to the eligibility screener survey.

Metrics

Metrics were compiled for each study’s online recruitment campaign per ad platform and, where data were available, per ad set. The metrics reported are:

• Impressions: the number of times an ad is viewed by a user on an advertising platform. Ads can be seen multiple times by the same user and each view counts as an impression;
• Clicks: the number of times an ad is clicked. Users can click on ads multiple times;
• Cost per click (CPC): the amount of money spent per click obtained in a given ad campaign or ad set;
• Click-through rate (CTR): % clicks/impressions;
• Number and % (of clicks) who began screener;
• Number and % (of those who began screener) who screened eligible;
• Number and % (of those eligible) who provided contact information for follow-up;
• Cost per eligible contact: the amount of money spent per eligible contact obtained through a given ad campaign or ad set.

Data on impressions and clicks were obtained from the advertising platforms and data on screener initiation through providing contact information were obtained from the SurveyGizmo data collection instrument (Figure 1). The cost per eligible contact was calculated as an overall measure of ad campaign effectiveness. Data on the recruitment source of enrolled participants is not available because the datasets of eligible contacts and enrolled participants could not be linked (this is due to the nature of the iTech network structure, in which the data center that administers eligibility screeners is separate from participant management staff, and thus participant IDs are not able to be transferred between screening and enrollment in an accurate and non-duplicative manner). Finally, the proportions of eligible contacts who were not non-Hispanic white and who were under the age of 18 were calculated, by platform and study (except P3, where race/ethnicity data was not captured in the eligibility screener).

Other recruitment efforts

In addition to recruiting through advertising on the aforementioned platforms, studies also conducted recruited through various in-person and community outreach efforts. These included posting flyers, tabling at community events, reaching out to past study participants and recruiting through clinics serving youth. Potential participants reached through these methods were directed to the same eligibility screeners as those reached through social media advertising. Basic metrics by study, including the number screened, the number who screened eligible, the number who provide contact information for follow-up and the proportions of eligible contacts who were not non-Hispanic white and who were under the age of 18, were calculated for the same timeframe as those for online advertising campaigns.

Results

Comparisons of metric by ad platform

Summary recruitment metrics per study and platform are presented in Table 2. The highest CTRs were observed for ads run on Grindr (7.54%) and Snapchat (1.32%). These ads also had the lowest corresponding CPCs ($0.20 and $0.15, respectively). However, these ads had the lowest proportions of users who initiated the eligibility screeners (1.0% and 2.1%, for Grindr and Snapchat respectively). Despite lower CTRs and higher CPCs as compared to ads on Grindr and Snapchat, Facebook/Instagram ads yielded the lowest cost per eligible contact across studies. When comparing ads that ran exclusively on Instagram with those that ran on all placements available for Facebook properties, Instagram ads had higher CPC and higher costs per eligible contact obtained. Twitter, which was used only for the P3 study, had lower CTR and higher CPC and cost per eligible contact when compared to its corresponding Facebook/Instagram recruitment campaign. Instagram yielded the highest proportions of eligible contacts who were non-white (70.6%) and under the age of 18 (35.3%). No eligible contacts under the age of 18 were recruited from Grindr.

Table 2 Ad performance and eligibility screener metrics by advertising platform for recruitment of young men who have sex with men into four different randomized control trials of HIV prevention interventions
Full table

Comparisons of metrics by study

The lowest cost per eligible contact was observed for ads recruiting for the MyChoices study across all platforms that were used for this study, while the highest cost per eligible contact was observed for ads recruiting for P3. Among those who screened eligible, the proportion who provided contact information for follow-up by study staff ranged between 57.7% and 100%.

Comparisons of metrics by geographic ad targets

Detailed metrics per geographic ad target for the MyChoices and LYNX studies are presented in Table 3. Both studies were RCTs of similar HIV prevention mobile apps and had virtually identical eligibility criteria besides city of residence. The Facebook/Instagram ad set that ran in North Carolina performed best across all metrics and was the only ad set with a CTR above 1% for the Facebook/Instagram platform. For the geographic targets (Chicago, Tampa, Boston, Bronx) that were consistent across platforms (Facebook/Instagram, Instagram, Snapchat), the cost per eligible contact was generally consistent, with ads in Boston and the Bronx generating the lowest cost per eligible contact as compared to Chicago and Tampa. The only exception was on Instagram, where Chicago had the lowest cost per eligible contact, though the numbers of those recruited through Instagram were low. The trends in other metrics (CTR, CPC, screening and eligibility rates) across cities were not as consistent, but in general, CPCs were higher for Chicago and Tampa than they were for Boston and Bronx. Screening and eligibility rates were lower for Chicago and Tampa than they were for Boston and Bronx. Grindr ads that ran in Chicago and Tampa were similar in terms of CTR, CPC, and proportion who began the screener. However, these ads produced low yields of eligible contacts.

Table 3 Ad performance and eligibility screener metrics by study site for recruitment of young men who have sex with men into two different randomized control trials of HIV prevention interventions
Full table

Comparison of social advertising recruitment and other recruitment efforts

Summary metrics by study and recruitment method (online advertising and other) are presented in Table 4. With the exception of P3, the number of those who entered the screeners through online advertising far exceeded those who were reached by other recruitment efforts. For all studies, the proportion of those screened eligible through online advertising was lower as compared to those screened eligible through other recruitment methods. However, the total numbers of eligible contacts banked were higher for online advertising campaigns for all studies except P3. Overall, the proportions of eligible contacts who identified as a racial or ethnic minority were lower for those recruited through online advertising as compared to other recruitment efforts, while the proportions of eligible contacts who were under the age of 18 was higher.

Table 4 Eligibility screener metrics for online advertising and other recruitment efforts four different randomized control trials of HIV prevention interventions for young men who have sex with men
Full table

Discussion

Recruitment was conducted for four studies between July 2018 and September 2019, using online advertising and other recruitment efforts such as in-person venue- and clinic-based outreach. This report adds to the literature by describing the comparative performance of different social media platforms and a dating/sex-seeking app for the recruitment of YMSM at high risk for HIV infection within different geographic areas of the US. Other researchers have reported the successful implementation of online advertising strategies to recruit YMSM for studies related to sexual health. Recently, Fontenot (30) described using advertisements placed on a dating app to recruit YMSM age 18–26 for web-based focus groups about a human papillomavirus (HPV) prevention app. Cost per enrollment ranged from $21 to $52. Another study that examined HPV prevalence among Australian YMSM recruited participants age 18–35 using Facebook advertisements and reported an average cost per enrollment of approximately $32 (31). Facebook advertising was also used to recruit gay, bisexual and/or queer-identifying males age 14–18 for an RCT of a text messaging-based HIV prevention intervention, with an average cost per enrollment of $17. These previously reported recruitment metrics are either lower or comparable to the lower end of the range of cost per eligible contact observed for iTech studies. However, it is important to note that iTech studies’ eligibility criteria (i.e., recent HIV risk behavior) and participation requirements (i.e., multiple in-person visits over time) present higher barriers to recruitment.

All four studies presented herein recruited HIV-negative YMSM. However, P3 had additional eligibility criteria that limited enrollment to those who recently initiated pre-exposure prophylaxis (PrEP) or reported imperfect adherence to PrEP. Unsurprisingly, this study had the highest cost per eligible contact and lower rates of screening and eligibility as compared to other studies that advertised on the same platforms. Recruitment for studies with stricter eligibility will generally cost more, especially when an eligibility criterion is not readily amenable to online targeting (e.g., medical information such as PrEP use or HIV status). Studies with such eligibility criteria that do not map well to ad targeting should consider using online recruitment as a supplement to in-person venue- and recruitment through clinics, which can identify potentially eligible participants among their existing clients based on medical history. Indeed, P3 was the only study where these in-person efforts yielded a higher number of eligible contacts than did online advertising.

Ads were placed on a variety of social media and networking platforms. When examining ad platform metrics alone, it would appear that ads on Grindr and Snapchat had markedly higher CTRs and lower CPCs than ads on Facebook properties. However, participants who clicked Facebook/Instagram ads had much higher rates of screening and ultimately yielded much lower costs per eligible participant. Additionally, ad design differences across platforms may influence clicks more than user interest in the advertised study. Ads on Grindr and Snapchat are shown as full screen images or videos between other content, so users may mistakenly click the ad when they mean to dismiss it. Furthermore, the audience targeting options available for Grindr and Snapchat are not as refined as those provided by Facebook; Grindr does not allow for ad targeting by age and Snapchat does not have any explicitly LGBTQ audiences predefined. Thus, the alignment between the ad audience and the studies’ eligibility criteria was not optimal for ads placed on these platforms. Although the advertising creative does explicitly and implicitly communicate the intended audience through the choice of images and ad copy, it is inevitable that ineligible users will click on the ads. Delivering ads to an audience with a higher proportion of ineligible users will result in more of those clicks. This could affect both screening and eligibility rates, as ineligible users could abandon the screener after seeing the study description on the landing page or they may continue through and screen ineligible.

Recruitment metrics were compared for ad campaigns that ran on all available placements on Facebook (including Instagram) and campaigns that ran exclusively on Instagram. The cost per eligible participant were markedly higher for the Instagram ad campaigns. Though Instagram has a higher share of younger users as compared to Facebook (2), the overall audience size is smaller which leads to more competition in the ad auction and therefore higher costs. Interestingly, the screening rate for users who clicked on Instagram-specific ads was higher than for those who clicked on combined Facebook/Instagram ads, but the eligibility rate was lower, despite the fact that both ad campaigns used nearly identical targeting options and creative. Thus, it may be that Instagram users have a different demographic, sexual risk and protective health behavior profiles than do Facebook users. Indeed, we observed that the highest proportions of eligible contacts who were not white and were under the age of 18 were recruited through Instagram.

Overall, ads placed on Facebook properties had the lowest overall cost per eligible contact recruited while ads on Twitter had the highest. However, Twitter was only used to advertise for P3, the study with the most stringent eligibility criteria. Despite the cost differences between the various advertising platforms, online recruitment for research studies may benefit from a multi-platform advertising strategy. Users may be more responsive to ad creative or branding that they see on multiple sites and generally, users need to be exposed to an ad multiple times before they are persuaded to take an action (32,33), such as screening for a study.

We compared ad performance across geographic targets for LYNX and MyChoices, as ad campaigns ran concurrently and their eligibility criteria were identical except for cities of residence. Ads that ran in North Carolina performed best across all metrics and had the lowest cost per eligible contact among ads that ran on Facebook/Instagram. Across all social media ad platforms, ad sets that ran in the Bronx and Boston generated lower costs per eligible participant than those that ran in Chicago and Tampa. This trend appears to be driven by both the advertising costs (CPC) and the screening and eligibility rates. There are many possible reasons for this. First, social media platforms charge per ad click based on an algorithm that considers audience size and competition from other ad buyers for the same audience. Thus, advertising costs will be higher in smaller media markets and areas where there is high demand from other advertisers for similar audiences. Second, different cities may have varying levels of community trust in research institutions or saturation for study recruitment, which may influence an individual’s decision to complete an eligibility screener for a study. Finally, geographic variations in HIV testing rates and sexual risk behaviors may affect eligibility.

Online advertising efforts in aggregate were compared to other recruitment efforts (in-person and community outreach), such as through clinics and community outreach. Though eligibility rates were generally greater for those screened through recruitment efforts (likely due to better targeting of potentially eligible participants), online advertising efforts yielded greater numbers of eligible contacts through higher screening volume. While it is not possible to calculate a cost per eligible contact for in-person and community outreach recruitment efforts, it is likely that online advertising was more cost-effective given the time and staffing effort required for these activities. The P3 study was the only exception to these trends, where both the number of individuals screened and the number of eligible contacts was higher for recruitment efforts. Furthermore, the proportion of non-white eligible contacts recruited through online advertising was lower than that of in-person recruitment, consistent with previous studies (34,35). However, a higher proportion of those recruited through online advertising were under the age of 18. This emphasizes the value of a diversity of approaches, as in-person and community outreach recruitment efforts may reach potential participants who may not be as easily targeted or responsive to online methods, and who have a different demographics than those recruited online.

While online advertising successfully recruited participants for several research studies, a number of challenges and limitations are noted. First, the algorithms that platforms use to target ads are not transparent to advertisers and it can be difficult to effectively match audiences with eligibility criteria. Further, platforms change their policies around ad content and targeting with little notice, which can be disruptive to study recruitment. A notable example of this is when Snapchat, midway through the recruitment period for LYNX, MyChoices and Get Connected, instituted a ban on advertising research studies to those under the age of 18. Second, ads that ran on Facebook and Instagram were open to comments from anyone who viewed them. A number of homophobic comments were posted which may have discouraged potential participants from engaging with the advertisements. While we did moderate these comments in order to remove discriminatory or bullying content (36), it is difficult to do this in real-time and the comments may have been seen regardless. Third, collecting contact information from someone who screened eligible does not guarantee that that individual will enroll as an RCT participant. Study staff reported high levels of non-response when attempting to follow up with eligible individuals to schedule them for baseline study visits. It is possible that participants recruited through in-person methods (e.g., reaching out to clinic patients or tabling at community events) may be more responsive to follow-up due to an established personal connection. However, our online recruitment efforts yielded more eligible contacts than did in-person and community outreach recruitment efforts. Fourth, our metrics may not be reproducible due to high variability in the digital marketing space. Ad performance is influenced by many factors including but not limited to ad content and creative, competing advertisements within platforms, and emerging platforms that may draw users to new digital spaces. However, the data presented herein serve as a useful general framework for evaluating the effectiveness of future digital advertising campaigns. Finally, the online methods used here produced a convenience sample that may not be representative of all YMSM.

There are several avenues for further inquiry to advance online recruitment methods. The effects of images and text on ad performance were not explored here and future work could focus on evaluation of different ad content aimed at promoting study trial participation among youth (10,30). The effects of landing page design and content should also be evaluated, especially given that sizeable drop-off occurs at the eligibility screener. It is possible that in-person or venue-based outreach allows recruiters to establish trust in ways that are not possible through an online ad, and thus are able to recruit participants who could be otherwise missed through online advertising. Future work could examine differences in youth recruited through online and in-person methods. Finally, effective recruitment of other sexual minority populations, beyond cisgender MSM, warrants attention. Recruitment for Techstep, an RCT for high-risk HIV-negative trans feminine, trans masculine, and gender non-conforming youth, launched recently using online advertising methods similar to those described herein and results will further the evidence base for recruitment of this key population (15).

Conclusions

We implemented a successful online recruitment strategy to recruit YMSM at high risk for HIV infection into four RCTs of mHealth interventions. Data from several concurrent advertising campaigns was compiled to fully characterize the ad launch to eligible contact recruitment cascade. This report provides a framework for evaluation of future online recruitment efforts across platforms and geographic areas, regardless of inevitable changes in the digital marketing space.

Acknowledgments

Funding: This work was supported by the National Institutes of Health Adolescent Medicine Trials Network for HIV/AIDS Interventions and the University of North Carolina/Emory Center for Innovative Technology (1U19HD089881).

Footnote

Provenance and Peer Review: This article was commissioned by the Guest Editor (Lisa Hightow-Weidman) for the series “Technology-based Interventions in HIV Prevention and Care Continuum among American Youth” published in mHealth. The article has undergone external peer review.

Data Sharing Statement: Available at http://dx.doi.org/10.21037/mhealth-20-64

Peer Review File: Available at http://dx.doi.org/10.21037/mhealth-20-64

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/mhealth-20-64). The series “Technology-based Interventions in HIV Prevention and Care Continuum among American Youth” was commissioned by the editorial office without any funding or sponsorship. LHW served as the unpaid Guest Editor of the series. LHW reports grants from NICHD, during the conduct of the study. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. All procedures were approved by the University of North Carolina at Chapel Hill institutional review board.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Centers for Disease Control And Prevention. HIV Among Youth. Centers for Disease Control and Prevention. Available online: https://www.cdc.gov/hiv/group/age/youth/index.html
2. Teens, Social Media & Technology 2018: Pew Research Center. Available online: https://www.pewresearch.org/internet/2018/05/31/teens-social-media-technology-2018/
3. Villanti AC, Johnson AL, Ilakkuvan V, et al. Social Media Use and Access to Digital Technology in US Young Adults in 2016. J Med Internet Res 2017;19:e196 [Crossref] [PubMed]
4. Sanchez T, Smith A, Denson D, et al. Internet-based methods may reach higher-risk men who have sex with men not reached through venue-based sampling. Open AIDS J 2012;6:83-9. [Crossref] [PubMed]
5. Hirshfield S, Remien RH, Humberstone M, et al. Substance use and high-risk sex among men who have sex with men: a national online study in the USA. AIDS Care 2004;16:1036-47. [Crossref] [PubMed]
6. Grov C, Cain D, Whitfield TH, et al. Recruiting a U.S. national sample of HIV-negative gay and bisexual men to complete at-home self-administered HIV/STI testing and surveys: Challenges and Opportunities. Sex Res Social Policy 2016;13:1-21. [Crossref] [PubMed]
7. Grov C, Westmoreland D, Rendina HJ, et al. Seeing Is Believing? Unique Capabilities of Internet-Only Studies as a Tool for Implementation Research on HIV Prevention for Men Who Have Sex With Men: A Review of Studies and Methodological Considerations. J Acquir Immune Defic Syndr 2019;82:S253-60. [Crossref] [PubMed]
8. Grov C, Breslow AS, Newcomb ME, et al. Gay and bisexual men's use of the Internet: research from the 1990s through 2013. J Sex Res 2014;51:390-409. [Crossref] [PubMed]
9. Ramo DE, Prochaska JJ. Broad Reach and Targeted Recruitment Using Facebook for an Online Survey of Young Adult Substance Use. J Med Internet Res 2012;14:e28 [Crossref] [PubMed]
10. Reiter PL, Katz ML, Bauermeister JA, et al. Recruiting Young Gay and Bisexual Men for a Human Papillomavirus Vaccination Intervention Through Social Media: The Effects of Advertisement Content. JMIR Public Health Surveill 2017;3:e33 [Crossref] [PubMed]
11. Wozney L, Turner K, Rose-Davis B, et al. Facebook ads to the rescue? Recruiting a hard to reach population into an Internet-based behavioral health intervention trial. Internet Interv 2019;17:100246 [Crossref] [PubMed]
12. Guillory J, Wiant KF, Farrelly M, et al. Recruiting Hard-to-Reach Populations for Survey Research: Using Facebook and Instagram Advertisements and In-Person Intercept in LGBT Bars and Nightclubs to Recruit LGBT Young Adults. J Med Internet Res 2018;20:e197 [Crossref] [PubMed]
13. Prescott TL, Phillips Ii G, DuBois LZ, et al. Reaching Adolescent Gay, Bisexual, and Queer Men Online: Development and Refinement of a National Recruitment Strategy. J Med Internet Res 2016;18:e200 [Crossref] [PubMed]
14. Burgess JD, Kimble RM, Watt K, et al. The Adoption of Social Media to Recruit Participants for the Cool Runnings Randomized Controlled Trial in Australia. JMIR Res Protoc 2017;6:e200 [Crossref] [PubMed]
15. Hightow-Weidman LB, Muessig K, Rosenberg E, et al. University of North Carolina/Emory Center for Innovative Technology (iTech) for Addressing the HIV Epidemic Among Adolescents and Young Adults in the United States: Protocol and Rationale for Center Development. JMIR Res Protoc 2018;7:e10365 [Crossref] [PubMed]
16. Ingram M. How Google and Facebook Have Taken Over the Digital Ad Industry: Fortune. Available online: https://fortune.com/2017/01/04/google-facebook-ad-industry/
17. Goldman R. Hard Questions: What Information Do Facebook Advertisers Know About Me? Facebook. Available online: https://about.fb.com/news/2018/04/data-and-advertising/
18. How Much Does Facebook Advertising Cost in 2019? WebFx, 2019. Available online: https://www.webfx.com/social-media/how-much-does-facebook-advertising-cost.html
19. Ford KL, Albritton T, Dunn TA, et al. Youth Study Recruitment Using Paid Advertising on Instagram, Snapchat, and Facebook: Cross-Sectional Survey Study. JMIR Public Health Surveill 2019;5:e14080 [Crossref] [PubMed]
20. Bauermeister JA, Golinkoff JM, Horvath KJ, et al. A Multilevel Tailored Web App-Based Intervention for Linking Young Men Who Have Sex With Men to Quality Care (Get Connected): Protocol for a Randomized Controlled Trial. JMIR Res Protoc 2018;7:e10444 [Crossref] [PubMed]
21. Liu A, Coleman K, Bojan K, et al. Developing a Mobile App (LYNX) to Support Linkage to HIV/Sexually Transmitted Infection Testing and Pre-Exposure Prophylaxis for Young Men Who Have Sex With Men: Protocol for a Randomized Controlled Trial. JMIR Res Protoc 2019;8:e10659 [Crossref] [PubMed]
22. Biello KB, Marrow E, Mimiaga MJ, et al. A Mobile-Based App (MyChoices) to Increase Uptake of HIV Testing and Pre-Exposure Prophylaxis by Young Men Who Have Sex With Men: Protocol for a Pilot Randomized Controlled Trial. JMIR Res Protoc 2019;8:e10694 [Crossref] [PubMed]
23. LeGrand S, Knudtson K, Benkeser D, et al. Testing the Efficacy of a Social Networking Gamification App to Improve Pre-Exposure Prophylaxis Adherence (P3: Prepared, Protected, emPowered): Protocol for a Randomized Controlled Trial. JMIR Res Protoc 2018;7:e10448 [Crossref] [PubMed]
24. Gebel M. In 15 years Facebook has amassed 2.3 billion users — more than followers of Christianity: Business Insider, 2019. Available online: https://www.businessinsider.com/facebook-has-2-billion-plus-users-after-15-years-2019-2
25. Facebook. Facebook Ads Manager. Available online: https://www.facebook.com/business/tools/ads-manager
26. Carman A. Snap is slowly growing, but it’s banking on augmented reality to sustain it: The Verge, 2019. Available online: https://www.theverge.com/2019/10/22/20927521/snap-growth-earnings-ceo-user-ar-augmented-reality
27. Snapchat. Snapchat Ads Manager. Available online: https://ads.snapchat.com
28. Bucksense. Grindr Ads. Available online: https://selfservice.grindr.com/login
29. Twitter. Advertise on Twitter. Available online: https://ads.twitter.com/login
30. Fontenot HB, Abuelezam NN, Rosenberger JG, et al. The Impact of Advertisement Messaging on Enrollment of Young Men Who Have Sex With Men for Web-Based Research: Observational Study. J Med Internet Res 2020;22:e16027 [Crossref] [PubMed]
31. Das R, Machalek DA, Molesworth EG, et al. Using Facebook to Recruit Young Australian Men Into a Cross-Sectional Human Papillomavirus Study. J Med Internet Res 2017;19:e389 [Crossref] [PubMed]
32. Effective Frequency: Reaching Full Campaign Potential: Facebook, 2016. Available online: https://www.facebook.com/iq/articles/effective-frequency-reaching-full-campaign-potential
33. Okadar G. How Frequency of Exposure Can Maximise the Resonance of Your Digital Campaigns: The Nielsen Company, 2017. Available online: https://www.nielsen.com/au/en/insights/article/2017/how-frequency-of-exposure-can-maximise-the-resonance-of-your-digital-campaigns/
34. Sullivan PS, Khosropour CM, Luisi N, et al. Bias in online recruitment and retention of racial and ethnic minority men who have sex with men. J Med Internet Res 2011;13:e38 [Crossref] [PubMed]
35. Reisner SL, Conron K, Scout N, et al. Comparing In-Person and Online Survey Respondents in the U.S. National Transgender Discrimination Survey: Implications for Transgender Health Research. LGBT Health 2014;1:98-106. [Crossref] [PubMed]
36. Russomanno J, Patterson JG, Jabson Tree JM. Social Media Recruitment of Marginalized, Hard-to-Reach Populations: Development of Recruitment and Monitoring Guidelines. JMIR Public Health Surveill 2019;5:e14886 [Crossref] [PubMed]