Methods, system errors, and demographic differences in participant errors using daily text message-based short message service computer-assisted self-interview (SMS-CASI) to measure sexual risk behavior in a RCT of HIV self-test use

Introduction

Sexual risk behavior assessment (SRBA) is crucial to HIV prevention research. To determine the effectiveness of behavior change interventions or biomedical prevention strategies, it is necessary to be precise about which sexual behaviors took place and what kind of protection was (or was not) used. This involves methods to observe, describe, explain, and predict behaviors that increase risk of adverse sexual health related events (1). Given that direct observation of human sexual behavior for research purposes is often both unethical and impractical, researchers must rely on indirect methods, such as collecting retrospective recall information through surveys or using biomarkers that may attest that a behavior took place. However, recall may be biased by memory inaccuracies, social desirability (the participant’s desire to please the researchers), and denial, among other reasons. Biomarkers (e.g., breathalyzers, hair strand tests and urinalysis to measure drug use) are often limited in terms of the period covered by the laboratory tests or their inability to discriminate sexual acts that took place protected from those that were not protected. Furthermore, these methods may be highly time consuming and expensive (2).

To decrease social desirability and reticence to report face-to-face sexual behavior that is often stigmatized, computer assisted self-interviews (CASI) are frequently used (3-7). Yet, CASIs that retrospectively cover extended periods of time may be inaccurate and lead to round up and other imprecisions. Administration of CASI assessments close to the actual time the behavior took place may improve the quality of the data collected. Furthermore, CASI administration via short message service (SMS, or text messaging), at present a ubiquitous form of mobile communication challenged by response errors (8-12), may constitute a significant step towards measurement improvement. Although our text message-based short message service computer-assisted self-interview (SMS-CASI) has been successfully used in other studies (13-15), it requires careful tailoring to study design and participants characteristics in order to be effective. Furthermore, considerations, such as privacy and security still exist when using a mobile phone to report sexual behavior (16).

With these considerations in mind, we used daily SMS-CASI in a confidential manner to assess sexual risk behavior in a study of HIV self-testing by non-monogamous sexual partners over six months (intervention group) and three months (control group). Here we present methods on how we implemented our SMS-CASI system and details on the tools we used to develop it. As a counter narrative to the higher error rates of participants using SMS-based data collection systems we present an analysis of our systems error rates, and demographic differences in participant errors when using our daily SMS-CASI to measure sexual risk behavior and follow study protocol. Finally, we go beyond stating that participant errors were frequent, which is typical of the data, and we provide use frequencies and metadata to provide a detailed picture of SMS-CASI use and feasibility of use by participants at a high reporting interval (i.e., daily) answering multiple questions.

Methods

Background

The study’s field name was iSUM (“I’ll show you mine”), a pun on the idea of potential sexual partners showing each other their HIV self-test results. This study was a 5-year randomized controlled trial exploring the effectiveness of HIV self-testing (HIVST) as a risk reduction tool for high-risk populations. Participants were recruited in New York City and Puerto Rico. Recruitment started in late March 2014 and participant follow-up was completed in December 2018. Participants were recruited through mixed-methods using social media-based tools and traditional outreach strategies to recruit high-risk and hard-to-reach populations. A detailed description of recruitment strategies has been previously published (17).

Participants were recruited based on the following criteria: HIV-negative; 18 years of age or older; identifying as a cisgender man or transgender woman who has sex with men; reporting three or more occasions of condomless anal intercourse (CAI) with serodiscordant or unknown status partners in prior three-months; two or more sexual partners in previous three months; and currently not on oral pre-exposure prophylaxis (PrEP). Participants respond to a brief pre-screening survey by phone or in-person. Those who qualify are invited to an in-person screening visit (Visit 1).

In Visit 1, participants completed a CASI to determine their eligibility for enrollment. The CASI assessed demographic information, sexual risk behavior in the past three months, HIV knowledge, alcohol and substance use history, and willingness to use a self-test with partners. Furthermore, participants self-administered an oral HIV antibody test using the (OraQuick^® Rapid HIV Test) while monitored by a researcher followed by a confirmatory test (Alere Determine™ HIV-1/2 Ag/Ab Combo Test) administered by staff. Those who fulfilled the eligibility criteria were invited to enroll in the study and asked to return for an enrollment visit (Visit 2).

In Visit 2, participants were randomized into either the control or intervention arm of the study and enrolled in the SMS-CASI (text messaging) system for daily reporting of sexual behavior and HIVST kit use. Participants in the intervention arm were given ten test kits to take home to self-test or use with potential partners; they could request refills during the three-month intervention period as needed. Participants in the control arm were not given test kits until after the three-month period, at which time they were given six kits to keep and use as needed. Participants were asked to return for a follow-up visit at month 3 (Visit 3), while those in the intervention arm were also asked to return for an additional visit at month 6 (Visit 4). In Visits 3 and 4, participants were tested for HIV and completed a CASI on-site, which assessed sexual behavior and the use of HIV self-tests with partners.

SMS-CASI system

The SMS-CASI system was programed using an internet based SMS sending and receiving service “Telerivet”, which provides a text message protocol design and development platform for SMS communications (https://telerivet.com/). SMS text messages were sent and received using a GSM text message gateway provider (https://www.nexmo.com/). Both a GSM SMS gateway provider and a Telerivet-based app were used. The app acted as a redundant system in case the GSM provider went down. Descriptions of both processes are described in detail on the Telerivet website.

At enrollment, participants were asked to bring their mobile phone and were trained to use the SMS-CASI system. During the training, they selected the time of day at which they preferred to receive the messages. They sent an initial “join” message to the system, and entered their desired reminder time and, subsequently, the password that was assigned to them based on their study participant ID (maximum 5 digits). After sending the password, participants were guided through an initial SMS-CASI reporting session, referring to a mock scenario to answer survey questions. Abbreviations used in the SMS-CASI script (see Figure 1, below) were explained by a study coordinator (AS = anal sex, C = condoms). Participants were also instructed how to get assistance, correct mistaken answers, change their reminder time, or change their phone number registration in the system. Finally, they were told that if they did not respond to the system for three days, they would be contacted by study staff to follow-up. Participants were given an instruction card to take home that included the compensation per session and the bonus, the password, the SMS-CASI system phone number, the selected reminder time, and study staff contact information.

Figure 1 SMS-CASI micro-questionnaire tree. SMS-CASI, short message service computer-assisted self-interview.

Per protocol, participants received daily text messages at a time of their choosing until the end of their participation in the study. Participants were first sent a reminder that asked about their readiness to report sexual behavior and HIVST kit use to the SMS-CASI system, which asked them to enter their password. They were then asked to report daily on the number of condomless anal intercourse (CAI) occasions since their last report, the number of those CAI occasions with a negative partner, the number of occasions in which they saw a partner’s negative test results, and the number of unused test kits remaining in their possession. Of note, by asking about number of CAI occasions “since last report,” participants were allowed to provide cumulative information for days in which reporting was missed. Intervention participants who reported having two or fewer kits in their possession were contacted to arrange for a resupply. Participants received a modest monetary compensation of one dollar for each completed SMS session, plus a 50% bonus if they reported an average of six out of seven sessions per week. Compensation was calculated prior to each follow-up visit from the total number of completed sessions during the 90-day follow-up period, and participants were remunerated at each follow-up visit (Visits 3 and 4) (see Figure 1).

All procedures were approved by the Institutional Review Boards at the New York State Psychiatric Institute and the University of Puerto Rico Medical Sciences Campus. The script for the micro-questionnaire tree was written in English and Spanish, and included a maximum of six possible messages sent and five possible messages received per day (see Figure 1). Text messaging data were monitored on a daily basis, including weekends. Study staff in New York reviewed the SMS-CASI system to determine and record the participants who did not report the previous day. Using a standardized electronic spreadsheet, study staff were able to record instances of non-responses for each study participant. Per protocol, participants who did not respond to the SMS-CASI system for three consecutive days were flagged for follow-up and were promptly contacted by a staff member from the corresponding research site via phone, email, or using other contact information provided by the participant.

Data cleaning and convergence

Data cleaning procedures

Prior to each follow-up visit (either a Visit 3 or Visit 4), a member of the study staff reviewed the daily SMS-CASI assessments to check for any anomalies in the data. Examples include: participant errors (e.g., sending an incorrect password, reporting more than 10 occasions of anal intercourse in a 24-hour period, messages unintentionally sent to the automated system, etc.), attempting to answer the same question multiple times, messages to study staff (e.g., “I need more kits”), and more than seven days of consecutive missed reports. In addition to identification of participants’ reporting errors, the data were reviewed for any system errors (e.g., errors in auto-calculations of participant compensation, the system sending the same question two or more times in a row, not preventing the participant from recording multiple reports on the same date).

Data convergence

In the event the SMS-CASI data contained errors related to questions 1–3 (see Figure 1), a calendar was created which compiled the participant’s daily SMS-CASI reports two weeks prior and two weeks post the errors (to give context and frame of reference to the reporting period in question). The calendar was presented to the participant during the follow-up session and the participant was provided the opportunity to correct any errors that were originally collected by the automated system. For example, if a participant missed ≥ seven consecutive reports, the calendar represented the two weeks prior to the missing days, the missing days, and the two weeks after the missing days. The participant then was able to either fill in the gaps if any unreported sexual activity occurred or confirm that the calendar was accurate, and no changes were needed.

System to identify data problems and ongoing monitoring of responses

Responses were monitored by SMS alerts programmed to be sent to research staff, in-real-time. For example, research staff were provided with a daily list of all participants who did not complete the SMS-CASI assessment each day. This enabled research staff to directly contact participants who may have been experiencing difficulty completing the reports and assist with any troubleshooting.

Procedures to clean data in real-time

An algorithm was created to begin cleaning participants' data in real-time. As participants navigated through the system, specific actions by the participants were automatically tagged with a label for further analysis. In addition, researchers reviewed all the SMS data at the end of each participant’s SMS reporting timeframe right before their final visit (both 3rd and 4th depending on study arm). Research staff manually tagged any message that fit into any of the participant or system errors outlined above (e.g., monetary compensation calculation errors, incorrect passwords). This was done for all participants and served multiple purposes. First, it was used to assess whether an SMS calendar needed to be generated and a convergence interview conducted. Second, any errors in participant compensation were corrected so that participants received accurate amounts of compensation at the conclusion of their follow-up appointment. Third, it significantly reduced the data management burden of cleaning and assessing the data for accuracy. Because this process was completed on an ongoing basis, it was not necessary to clean the data all at once at the conclusion of the study, thus facilitating data analysis to begin promptly after data-collection was completed.

Measures and data analysis

Descriptive statistics were used to examine frequencies related to participant reporting, response rates, communication patterns, engagement with the SMS-CASI system, study retention, and overall feasibility of using the SMS-CASI system for reporting sexual risk behavior. In addition, demographic data from participants’ baseline on-site CASI assessment, including age, race, ethnicity, employment/student status, and income were examined. Descriptive data were generated using SPSS Statistics* (*IBM Corp. Released 2016. IBM SPSS Statistics, Version 24.0. Armonk, NY: IBM Corp).

Results

Participant retention

A total of 1,419 individuals were pre-screened (over the phone for behavioral eligibility; 368 participants attended Visit 1 to determine negative HIV status and behavioral eligibility, collected via on-site CASI (see Figure 2). Those eligible to enroll (N=272) returned within 7 days for Visit 2, enrollment and randomization. After a three-month intervention period, 259 participants returned for Visit follow-up. Finally, 123 participants randomized to the intervention condition returned at six months for a final follow-up visit (Visit 4). Participant retention at the 3-month follow-up was high (95%), and remained relatively high at 6-month follow-up (90%).

Figure 2 Participant retention.

Characteristics of the participants enrolled in the study

The mean age of the sample was 34 years old (range, 18–69, SD =11.12) years. Most participants identified as male (N=245; 90%); However, some participants identified as transgender female (N=27; 10%). Similarly, most participants identified as gay or homosexual (N=212; 78%), with the remainder identifying as bisexual (N=42; 15%), straight/heterosexual (N=7; 3%) or Other (N=11; 4%). Employment among participants was high, with two-thirds employed (N=180; 66%), and a third unemployed. A few participants reported being students (N=49; 18%). Finally, participants' annual income had a wide range ($0–$220,000). Nevertheless, more than half of participants (N=138; 57%) reported income of less than $20,000 (see Table 1).

Table 1 General sample characteristics
Full table

SMS-CASI system

SMS-CASI descriptive data

Over the course of the study, the SMS-CASI system processed 272,565 messages. In addition, 87 classifications of metadata were collected. Types of messages included those sent (N=171,749; 63.01%) to participants, received (N=100,646; 36.93%) from participants, and failed (N=168; 0.06%) to be sent. The most useful types of metadata for this study included: Date, From, To, Contact Name, Message, Message Type, Direction, Status, Error Message, PPT Error, Time Sent, Parts, Call Duration, Starred, Labels, Compensation Error, Delayed Reply, Duplicate, Question Answers, Enrollment, First Report, Kit Increase, Message to Staff, System Error, Source, Service Name, User Account Number, Contact ID, Message ID; and Message Cost to name a few. Each message was presented in a row and each classification of data presented in a column; each intersection of messages with classifications represented a singular data (or metadata) point, for a total of 22,895,460 data points.

SMS-CASI system generated errors

There were 195,283 instances of labels that describe the nature, circumstance, or characteristics of each message. Errors were among the least common labels (N=1,858; 0.95%), meaning they represented a negligible fraction of all messages (0.68%). Of those errors, 34 (1.83%) were compensation errors where the system mis-estimated the compensation that was owed the participant. This was the result of a combination of factors: (I) the system allowing the participant to report more than once in a day; Thus, accruing more compensation than was allowed on a daily basis, and (II) the participant actually attempting to game the system and accrue more compensation than was allowed. This programming glitch was fixed, and the additional reports were labeled accordingly. Though compensation errors were a combination of participant action and a programming glitch, there were still errors that were solely the result of the system itself. Of the 1,858 errors labeled, there were 535 errors (28.79%; Label = “System Error”) that were due to system malfunctions.

Participant generated errors by demographics

In contrast to the SMS-CASI system, of the 1,858 errors generated over the course of the study, participants generated the most errors (N=1,289; 69.38%; Label = “PPT Error”). Examples of participant error types included: using the wrong password, not completing a prior micro-questionnaire, and sending a wrong or invalid response to the question asked. Participant generated errors were disproportionately higher among older participants with participants ages 18–24 generating 204 errors (15.83%), ages 25–29 generating 157 errors (12.18%), ages 30–39 generating 383 errors (29.71%), and ages 40–69 generating 545 errors (42.28%). In terms of Race/Ethnicity Hispanic/Latino participants generated the most errors at 487 errors (37.78%), followed by White 290 (22.50%), Black/African-American 375 (29.09%), and Asian 95 (7.37%) Native American 42 (3.26%). Level of education and error rate correlated with each category’s respective N, with the exception of HS Grad/GED, which was notably higher. Error rates were: Less than HS 36 (2.79%), HS Grad/GED 374 (29.01%), Partial College 286 (22.19%), College Grad/Grad School 593 (46.00%). Unpredicted disparities in number of errors also occurred with participants that made the least amount of income and the most amount of income: less than $10,000: 528 (40.96%), $10,000–$19,999: 142 (11.02%), $20,000–$29,999: 174 (13.50%), $30,000–$39,999: 116 (9.00%), $40,000–$220,000: 329 (25.52%). Demographics such as gender showed error rates commiserate with their respective recruitment, with men generating 1,196 errors (92.79%), and again Sexuality Identity showed similar error trends: Gay/Homosexual 941 (73.00%), Bisexual 271 (21.02%), Straight/Heterosexual 64 (4.97%), and Other 13 (1.01%). Some demographics such as Employment/Student Status showed no trends, and some showed small trends, but these categories were not mutually exclusive; thus, impossible to elucidate errors by category. Moreover, missing demographic data meant that categories like Race/Ethnicity and Levels of Education could add up to greater than 100% of the population N, and categories like Employment/Student Status could be less than the study N. However, most errors in each demographic category were generated by only a few participants in each category, with a range as few as two participants generating all errors in a sub-demographic category (i.e., Asian) in the Race/Ethnicity category, and as many as 187 participants generating at least one error from at least one error type in the Gender sub-category (i.e., Men), but only 37 participants generating over 90% of all errors. Lastly, SMS-CASI analysis by site (i.e., Puerto Rico vs. New York) are reported elsewhere.

SMS-CASI communication examples

Although communication between the participant and the SMS-CASI system was automated, participants sometimes sent non-standardized messages, using the SMS-CASI system as a direct line of communication with the research staff. Once received, staff members responded to the message manually. Messages ranged from requests from participants for assistance to comments on their experience in the study. Occasionally, researchers sent messages on updates to the system (see Figure 3).

Figure 3 Communication examples.

Discussion

The use of SMS as a real-time data collection tool is becoming more frequent in health behavioral research. Though some studies use SMS to contact their participants or ask them sequential questions, none have detailed the successful integration of SMS with CASI, except for one of our previous studies (13). Even in our own previous study, the micro-questionnaire was not developed with the kind of intelligent skip patterns shown in Figure 1. This study details the systematic design and implementation of SMS-CASI for a multi-site and dual-lingual national study. We found that what worked for a study to monitor medication use (i.e., pre-exposure prophylaxis), could be used to monitor the use of HIV self-tests among non-monogamous sexual partners. We also demonstrate an effective way of asking sexual health related questions that mitigates risk of accidental disclosure via the participants mobile phone and increases privacy and security.

Furthermore, standard methods of assessing sexual risk behavior typically involve delayed self-reports (18,19). Unfortunately, traditional self-report (i.e. retrospective interviews, focus groups, in-person surveys) introduces both recall bias and social desirability bias into the data. Reducing bias in self-reporting is one of the most challenging tasks in behavioral health research. Though, other studies like Lizama et al. observed a significant amount of recall bias in participants’ reports (18), in our study we attempted to mitigate recall bias by using daily SMS-CASI to collect self-reports of sexual risk behavior, in real-time. By shortening the window of recall and mitigate face-to-face social desirability with daily SMS-CASI, participants were able to tally sexual risk behavior closer to the time of the event without feeling the pressure of a researcher in front of them or in the next room waiting for a response.

In addition, using a simple repeated micro-questionnaire of less than five questions simplified the response process and facilitated quick reporting, high use, system acceptability, which we feel translated to and was exemplified by our high retention rates. The SMS-CASI also acted as a diary record of the participants’ histories. On several occasions, it was observed that participants were able to review their saved text messages when attempting to recall events during their final interview and when completing the more detailed final CASI at the study site.

In addition to describing the ways daily SMS-CASI could potentially reduce biases in self-reporting and contribute to retention, we also demonstrated how SMS-CASI is a highly effective and acceptable method of collecting behavioral data from participants. Participant engagement with the SMS-CASI system was high, as demonstrated by high system output, participant utilization, and additional communication patterns. Most participants actively engaged with the SMS-CASI system most of the time, with a large number of participants reaching engagement levels of greater than 80% and receiving a compensation bonus.

Error rates were diverse with older participants generating a disproportionate number of errors. Errors also varied by race with Hispanic/Latino generating the most errors, but none of the race/ethnicity categories generated errors that were greatly disproportionate to their sample size. Moreover, though most participants generated some errors, most errors were generated by a small number of participants. These few participants’ excessive error rates may indicate a type of hyper-engagement that was in some way unique rather than anomalous, and might warrant further examination. Another indication of participant engagement with the system was the fact that participants did not only use the system to complete the micro-questionnaire, but also to communicate directly with the research staff. Participants were not instructed to do this, but many attempted to communicate with the study site via the SMS-CASI system. Participants communicated problems, questions, and general feedback. Researchers responded to participants in kind. Once it was clear that the system was a desirable method of communication, it became standard to use the system to directly respond to questions and comments that were received via SMS.

Future work

Future work in this area should examine the effect of SMS-CASI on behavioral assessment more broadly and sexual risk behavior specifically. Additional research should look into the ways in which SMS-CASI can reduce biases and to what extent it can validate those results within standard statistical significance measures. Assessment of SMS-CASI as a data collection tool should be studied in a specifically designed control trial to evaluate the effectiveness of such technology in real world settings. One of the unexpected challenges that occurred during this study was the landfall and devastation of two hurricanes (i.e., Maria and Irene) at our Puerto Rico site. Though we report elsewhere about the effects of these hurricanes on our SMS-CASI system and our study as a whole, more research should look specifically into the use of SMS-CASI and disaster preparedness measures in extreme weather disaster prone research sites.

Limitations

This study was not originally designed to infer SMS-CASI’s effect on sexual risk behavior assessment. However, use of an SMS-CASI for daily automated assessment provided a rare opportunity to analyze participant sexual risk behavior. Here we only discuss SMS-CASI. Though SMS-CASI provided the primary outcome for the research, other data tools were used and are discussed elsewhere. Thus, the analysis of these results is limited to what can be inferred through only the SMS-CASI data. Also, this study was performed at only two sites with a very specific sample population. This level of sampling specificity limits the generalizability of the findings. Any reviews of the literature that inform this paper are also limited by the fact that we drew from English language publications only.

Conclusions

This study provides methodological examples and preliminary data to demonstrate how SMS-CASI could be used as a collection tool for sexual health data and potentially to mitigate common biases associated with self-report. The use of daily SMS-CASI in the context of using HIV home test kits with non-monogamous sexual partners was feasible. The high rates of system use, data output, and communication via the SMS-CASI system was indicative of high participant system utilization, which suggests high acceptability of using the system. In addition, daily SMS-CASI shortens more traditional recall time periods from weeks and months, to 24 hrs, and using our micro-questionnaire with five survey items or fewer simplifies the reporting process, making it easier for participants to acclimate to reporting since they have to recall fewer behaviors. Social desirability bias is most impacted by triggers related to human interaction. SMS-CASI minimized the periods of face-to-face human interaction, which potentially decreases participants' inclination to respond in a socially desirable way.

Acknowledgments

Funding: This work was supported by the Eunice Kennedy Shriver Institute of Child Health and Human Development and the National Institutes of Health [R01 HD076636 to ACD]. The HIV Center for Clinical and Behavioral Studies is funded by an National Institute of Mental Health center grant [P30-MH43520 to BR]. William Brown III was supported by the National Library of Medicine (NLM) [grant numbers R01-LM012355 to DS, T15-LM007079 to GH], and the National Institute on Minority Health and Health Disparities (NIMHD) grant number P60-MD006902 to KBD] during various stages of the research and/or preparation of the article. The content is solely the responsibility of the authors and does not necessarily represent the official views of NICHD, NIMH, NLM, NIMHD, or the NIH.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

Ethical Statement: All procedures were approved by the Institutional Review Board at the New York State Psychiatric Institute (IRB# 6854) and the University of Puerto Rico Medical Sciences Campus. Written informed consent was obtained from the participants for publication of this manuscript and any accompanying materials.

References

1. Duda SN, McGowan CC, Gadd CS. Perceived Reasons for High and Low Quality Observational HIV Research Data. Stud Health Technol Inform 2015;216:994. [PubMed]
2. Althubaiti A. Information bias in health research: definition, pitfalls, and adjustment methods. J Multidiscip Healthc 2016;9:211-7. [Crossref] [PubMed]
3. Fincher D, VanderEnde K, Colbert K, et al. Effect of face-to-face interview versus computer-assisted self-interview on disclosure of intimate partner violence among African American women in WIC clinics. J Interpers Violence 2015;30:818-38. [Crossref] [PubMed]
4. McNeely J, Strauss SM, Rotrosen J, et al. Validation of an audio computer-assisted self-interview (ACASI) version of the alcohol, smoking and substance involvement screening test (ASSIST) in primary care patients. Addiction 2016;111:233-44. [Crossref] [PubMed]
5. Spear SE, Shedlin M, Gilberti B, et al. Feasibility and acceptability of an audio computer-assisted self-interview version of the Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) in primary care patients. Subst Abus 2016;37:299-305. [Crossref] [PubMed]
6. Trick WE, Deamant C, Smith J, et al. Implementation of an audio computer-assisted self-interview (ACASI) system in a general medicine clinic: patient response burden. Appl Clin Inform 2015;6:148-62. [Crossref] [PubMed]
7. Vu LTH, Nadol P, Le LC. HIV-related risk behaviors among the general population: a survey using Audio Computer-Assisted Self-Interview in 3 cities in Vietnam. Asia Pac J Public Health 2015;27:NP798-807. [Crossref] [PubMed]
8. Iribarren SJ, Brown W, Giguere R, et al. Scoping review and evaluation of SMS/text messaging platforms for mHealth projects or clinical interventions. Int J Med Inform 2017;101:28-40. [Crossref] [PubMed]
9. Lauffenburger JC, Choudhry NK. Text Messaging and Patient Engagement in an Increasingly Mobile World. Circulation 2016;133:555-6. [PubMed]
10. de Lepper AM, Eijkemans MJC, van Beijma H, et al. Response patterns to interactive SMS health education quizzes at two sites in Uganda: a cohort study. Trop Med Int Health 2013;18:516-21. [Crossref] [PubMed]
11. La Rue EM, Li Y, Karimi HA, et al. A Description of the Development and Architecture of an SMS-Based System for Dealing With Depression. Procedia Technol 2012;5:670-8. [Crossref]
12. Haberer JE, Musiimenta A, Atukunda EC, et al. SMS reminders plus real-time adherence monitoring improve adherence to antiretroviral therapy in rural Uganda. AIDS 2016;30:1295-300. [Crossref] [PubMed]
13. Brown W, Giguere R, Sheinfil A, et al. Challenges and solutions implementing an SMS text message-based survey CASI and adherence reminders in an international biomedical HIV PrEP study (MTN 017). J Biomed Inform 2018;80:78-86. [Crossref] [PubMed]
14. Carballo-Diéguez A, Balán IC, Brown W III, et al. High levels of adherence to a rectal microbicide gel and to oral Pre-Exposure Prophylaxis (PrEP) achieved in MTN-017 among men who have sex with men (MSM) and transgender women. PloS One 2017;12:e0181607. [Crossref] [PubMed]
15. Balán IC, Giguere R, Brown W, et al. Brief Participant-Centered Convergence Interviews Integrate Self-Reports, Product Returns, and Pharmacokinetic Results to Improve Adherence Measurement in MTN-017. AIDS Behav 2018;22:986-95. [Crossref] [PubMed]
16. Giguere R, Brown W, Balán IC, et al. Are participants concerned about privacy and security when using short message service to report product adherence in a rectal microbicide trial? J Am Med Inform Assoc 2018;25:393-400. [Crossref] [PubMed]
17. Iribarren SJ, Ghazzawi A, Sheinfil AZ, et al. Mixed-Method Evaluation of Social Media-Based Tools and Traditional Strategies to Recruit High-Risk and Hard-to-Reach Populations into an HIV Prevention Intervention Study. AIDS Behav 2018;22:347-57. [Crossref] [PubMed]
18. Lizama N, Heyworth J, Thomson A, et al. Self-reported shift work, recall bias, and belief about disease causation in a case-control study of breast cancer. Cancer Epidemiol 2017;50:9-15. [Crossref] [PubMed]
19. Cherpitel CJ, Ye Y, Stockwell T, et al. Recall bias across 7 days in self-reported alcohol consumption prior to injury among emergency department patients. Drug Alcohol Rev 2018;37:382-8. [Crossref] [PubMed]