Development and usability of a Spanish/English smoking cessation website: lessons learned

Introduction

Online interventions have been found effective in helping smokers quit (1,2). However, there is a dearth of online interventions in Spanish. Earlier studies by our group (3,4) showed that Spanish/English smoking cessation Internet interventions could yield abstinence rates comparable to those reported for the nicotine patch. Although English-speaking smokers were twice as likely as Spanish-speaking smokers to have used smoking cessation aids prior to visiting our previous sites (5), the abstinence rates for both language groups were very similar. Indeed, the 12-month missing = smoking (participants who cannot be reached at follow-up are presumed to be smoking) abstinence rates were 20.2% for Spanish speakers and 21.0% for English speakers (4). The stop smoking website that was used reduced disparities in access to evidence-based smoking cessation interventions, and comparable outcomes in both groups suggested it was effective regardless of language or history of smoking cessation aids.

However, because the intervention was provided online, it required computer literacy and access to the web. Thus, few smokers with low levels of education joined these studies, and individuals who rated themselves lower in the socioeconomic continuum and those from lower-income countries reported significantly lower abstinence rates than better-off individuals from more affluent countries. The wealthier, more educated smokers living in wealthier countries benefited more from the intervention than less wealthy, less educated smokers living in poorer countries (5).

The aim of this study was to evaluate modifications to online smoking cessation interventions that would increase engagement from Spanish- and English-speaking smokers. In addition, the study aimed to reach low-income smokers; to increase the likelihood of doing so, we interviewed primary care patients of the San Francisco Health Network (SFHN) and obtained suggestions regarding what they would find useful on a smoking cessation web app. This report describes the process of developing a Spanish/English web app across three versions, the eventual outcomes, and the lessons learned. We will present the characteristics of the three versions of the smoking cessation site and the utilization metrics associated with each of them.

Methods

Version 1

Goal

To create a basic smoking cessation web app that contained the key elements found to be effective in earlier studies. By starting with a basic app and adding more functionality in later versions, it was possible to examine the relative effectiveness of various elements vis-a-vis utilization and engagement. Version 1 was active from January 12, 2016 to February 21, 2017, recruiting 64 smokers [n=26 (41%) Spanish-speaking and n=38 (59%) English-speaking] eligible to enter the intervention by verifying their email address.

Developers

Version 1 of the web application was developed by the research team and custom-coded by an external team according to the specs provided. Twilio, a communication platform that allows users to send and receive text messages or other forms of communication using API (i.e., application program interface) code, was utilized to send SMS messages.

Formative user feedback

Interviews with public sector primary care patients were conducted to guide the development of Version 1. A total of 24 (18 English-speakers, 6 Spanish-speakers) low-income smokers participated in these interviews. Although a majority of the participants owned smartphones (purchased or a government issued), mobile literacy was low. For example, many of those interviewed did not know how to go to a specific site in their web browser or sometimes, how to open the browser. Among the reasons for quitting smoking, health was the most important reason. Some participants reported wanting to quit smoking because of their families and/or children. However, participants reported having few coping skills outside of smoking. None had used an app to quit smoking, although some people reported having used the internet to obtain health information and/or were willing to try a web app.

Content

Basic content for the smoking cessation website was based on a review of best practices for internet intervention developments and prior experience with such interventions (3,4). Consistent with the goal for Version 1, this version of the web app was relatively austere and included elements that are common to most self-help smoking cessation apps. Specifically, information from the “Guía Para Dejar de Fumar” (Stop Smoking Guide) (6) was included, which listed health risks related to smoking and strategies to quit smoking, such as not buying cigarette packs, asking smoking friends not to share their cigarettes, and learning how to refuse cigarettes when offered. Version 1 also provided tools to track cigarettes smoked and to set a quit date.

Text messages

Participant phone numbers were required in Version 1 to send text message reminders for follow-up surveys. Phone numbers were only utilized for the follow-up reminder texts.

Appearance

The appearance of Version 1 was purposefully simple, with a plain background image and a straight-forward organization of elements on each page (see Figure 1). The onboarding process of Version 1 utilized an external survey form; users were directed to the survey to complete onboarding and redirected back to the web app after completion.

Figure 1 Sample screenshot of version 1.

Version 2

Goal

To include suggestions from the public sector smoker interviews in the new version. This version was active for 8 months, from February 28, 2017 to October 23, 2017. During this time, the automated website recruited 97 (30%) Spanish-speaking and 227 (70%) English-speaking participants, for a total of 324 smokers.

Developer

As with Version 1, content and design elements were created by the research group and programmed by the same external development team.

Formative user feedback

We conducted field studies in which we “shadowed” primary care patients who smoked. A total of 24 participants were involved. Initially, observational and interview-based studies on their smoking behavior and smartphone use were conducted. Later, participants were asked to utilize the Version 1 web app, and, after a week of use, interviews to document their reactions to the web app were scheduled. Finally, two groups of participants from SFHN (English-speaking n=18, Spanish-speaking n=6) met to consult with the research team on the design of the web app. The information collected from these meetings was intended to influence the further development of the web app. The feedback from the primary care patients generally suggested that the application be more visually interesting and more “fun”. Feedback also suggested the removal of barriers to participants joining the app, such as removing the requirement to confirm their email and phone number or reducing the length of the onboarding process.

Content

Version 2 contained all elements of Version 1, and also added several others: relaxation recordings, a quit plan, and smoking cessation tips, as well as mood, craving, and money trackers. The quit plan guided new users through the web app’s features, with the goal of enticing users to use all of the elements. Based on interviewees’ suggestions, smoking cessation tips were added to engage users in developing coping skills outside of smoking (e.g., “Get enough sleep today. Being well-rested can help you achieve your goals.”) and reducing cigarettes smoked in the days leading up to their quit date (e.g., “Practice resisting cravings today. Try to go 5 minutes without lighting up.”). Participants could rate helpfulness of each tip. Since participants could track data regarding their smoking, cravings, mood, and money, Version 2 also included a page that summarized their data. Mood tracking was added because earlier studies had shown that helping smokers manage their mood increased quit rates (7).

Text messages

In response to feedback given for Version 1, Version 2 expanded the text message feature significantly to include regular text messages to notify participants about their quit date and encourage participants in their progress. Text messages that were sent surrounding a user’s quit date were used to alert them that their quit date was coming up (e.g., “You have 2 weeks to go before your quit date.”) or celebrating their success (e.g., “You’ve been smoke free for 2 weeks. Your circulation should improve, exercising should become easier and lung function should increase by 30%.”). If users indicated that they had smoked following their quit date, they received a text message encouraging them to try again (e.g., “Many people slip a few times before quitting for good. Remember, many difficult things take several attempts. Think about difficult things you’ve gotten through in the past.”). For users who did not set a quit date, text messages were focused on encouraging participants to try quitting by utilizing their stated quit reasons. For example, if a user indicated that they wanted to quit for health reasons, they got texts with information on the health detriments of smoking (e.g., “Smoking is the main cause of lung cancer, but did you know it can also cause cancer in the mouth, esophagus, throat, stomach, pancreas, bladder and cervix?”). Follow-up reminders were also sent via text message with a url to the web app, as in Version 1.

Appearance and user experience

The visual design for Version 2 was changed significantly. A simple design was retained, but icons and buttons were redesigned to improve visibility and appeal. Additionally, the layout of the homepage was improved to include a daily tip and the participant’s quit date (see Figure 2). The onboarding process was integrated into the web app instead of using an external survey form to make a more seamless experience for users. Additionally, the onboarding questionnaire was shortened to facilitate a faster entry into the web app features.

Figure 2 Sample screenshot of version 2.

Version 3

Goal

To build on Version 2 with a special focus on design and user experience; specifically, to improve the ease of use and visual appeal. This version was extensively redesigned based on user data, several new features were added, and less useful features from past versions were removed. Version 3 actively recruited smokers for approximately 7 months, from October 24, 2017 to May 25, 2018. During this period, we recruited 508 (46%) Spanish speakers and 599 (54%) English-speakers, for a total of 1,107 participants.

Developer

Unlike Versions 1 and 2, which were designed by our research team but developed by external programmers, Version 3 was developed by an internal developer, M.B., a doctoral student in clinical psychology with experience in web development and graphic design. Working with an internal developer was advantageous because it both streamlined communication as well as improved flexibility in the design/development process. Furthermore, the clinical psychology training allowed the developer to be a full participant in the intervention design process. Version 3 utilized Caspio, an out-of-the-box data collection software, to create the database. Wordpress.org was used to create a custom appearance with minimal coding. SMS messages were sent using Twilio.

Formative user feedback

No direct user feedback was sought for this version. Decisions about content changes (e.g., removing or bolstering certain features) were guided by the utilization data gathered from Versions 1 and 2. Design-related changes were guided by the principles and practices of graphic design.

Content

The content for Version 3 was primarily the same as the content for Version 2, with some changes based on feedback from consultants. Because previous versions exhibited so little engagement from participants, the main goal of Version 3 was to incentivize participants to engage more heavily and provide more tracking data. Data visualizations and summaries were included on the dashboard of the web app, which showed the participants their smoking, mood, and craving data over time. Data summaries presented the participant with snapshots of their progress or smoking habits, such as money saved through smoking less or their most commonly reported smoking trigger. Additionally, a trigger diary was added for smoking triggers, which presented the participant with their entire cigarette tracking history and the triggers they identified for each smoking event. Improvements were made to the content of the tips given to participants to improve clarity, and the ability to rate a tip was replaced with an ability to mark a tip as a “favorite” (favorite tips could be viewed together at any time). To further simplify the site, the quit plan introduced in Version 2 was also removed. A link to the interactive or PDF version of the Guía para Dejar de Fumar (our Smoking Cessation Guide) was included on the landing page for Version 3. Thus, every visitor was given access to our Smoking Cessation Guide in their chosen language without being required to sign up for the study.

Text messages

Text messages in Version 3 were used to notify participants about their quit date, encourage participants in their progress, remind participants to use the web app, and engage participants in follow-up questions about their smoking status. Although Version 2 included text message reminders and encouragements, texts in Version 3 placed a higher emphasis on user engagement with a weekly text, reminding participants to track data and texts notifying participants that they had not been active on the web app recently. Initially, follow-ups were sent using a link to the web app. After low response rates were observed early on, the follow-up questions were embedded directly into an SMS survey so that participants did not have to sign in to complete the follow-up survey. If the participant did not respond to the initial question on their follow-up day, the same text was sent the following day to further encourage a response.

Appearance and user experience

The web application was designed to be sleek and modern, with a simple color scheme, responsive sizing, and high-resolution images. To entice users to sign up to the site, a great deal of information about the web app was added to the landing page, including a summary of the application’s features, information about the research team, screenshots from within the web app, and reasons for quitting smoking.

Unlike Version 2, which had multiple pages of sign-up questions without a progress bar, Version 3 minimized the questions, and placed everything on one page. Thus, a participant interested in signing up was presented with a single page asking for email, and password, as well as some demographic information, with some optional questions, such as their phone number. After signing up, the participant was instructed to log in, at which point they were presented with a dashboard. The dashboard contained the trackers (cigarettes, mood, and craving), quit date and quit countdown, as well as graphs and summaries of the participant’s tracked data (see Figure 3). Participants could also access their trigger diary from the dashboard, which included raw data about the cigarettes they had tracked. To aid the participant in understanding the web app’s uses, a popup appeared to walk them through the major features.

Figure 3 Screenshot of version 3.

Ethical considerations

This study was approved by the Palo Alto University Institutional Review Board (15-042) as well as the University of California, San Francisco Committee on Human Research (15-17597). All participants completed an online consent form prior to engaging with any version of the web app.

Results

Participants/recruitment

Participant demographics are presented in Table 1. Recruitment strategies differed across versions and therefore the number of recruited participants cannot be directly attributed to the web app content or design. The final sample for Version 1 had 64 participants and was 56.7% male, 59.4% English-speaking, and 44.4% Hispanic/Latino. Age was not asked in Version 1, but users had to indicate they were over 18 to participate. Version 2 recruited 324 participants (M_age=37.1). The sample was 58% male, 70.1% English-speaking, and 48.1% Hispanic/Latino. The final sample for Version 3 had 1,107 participants (M_age=46.7) and was 39.3% male, 54.1% English-speaking, and 51.0% Hispanic/Latino.

Table 1 Demographics for All Versions
Full table

Engagement

Engagement data for all versions is presented in Table 2. Because the sample size of Version 1 is significantly smaller than the latter two versions, only direct comparisons between Version 2 and 3 are examined. Participants in Version 3 were significantly more likely to engage all three trackers. Specifically, 56.8% of all participants in Version 3 tracked their cigarette usage at least once compared to only 11.7% of Version 2 participants (χ²=204.79, P<0.0001). The mood tracker revealed a similar pattern with 34.4% of Version 3 participants and 12.7% of Version 2 participants using it (χ²=27.92, P<0.0001). The craving tracker was the least popular among both Version 2 and Version 3 participants, with 6.8% and 26.8% using it, respectively (χ²=92.32, P<0.0001).

Table 2 Engagement metrics for all versions
Full table

Text messages

The SMS messages sent out with the follow-up survey had varied success across Version 2 and Version 3 (see Figure 4). Within Version 2, 242 (74.7%) of 324 phone numbers were invalid and SMS messages could not be sent. Of the remaining 25%, SMS service errors and users stopping the SMS service affected who received the follow-up messages. Thus, the three follow-up SMS messages were received by 35 (11%), 46 (14%), and 51 (16%) participants, respectively. Within Version 3, phone numbers were not required and 170 (15%) of participants did not give a phone number. Additionally, invalid phone numbers and quit dates were given, participants stopped the text message services, and the SMS service experienced errors. After accounting for these issues, the three follow-up messages were received by 367 (33%), 450 (41%), and 441 (40%) participants.

Figure 4 Follow-up SMS success rates for version 2 and 3.

Follow-ups

To account for the large percentage of text messages that were not received in Version 2 and Version 3, follow-up rates for these versions use text messages successfully sent as the denominator. For the first follow-up survey, 16 (25%) of the 64 Version 1 participants responded; 3 (8.6%) out of 35 participants who received the survey SMS responded in Version 2; and 106 (28.9%) out of 367 participants responded in Version 3. For follow-up 2, 25 (39.1%) of 64 Version 1 participants, 2 (4.3%) out of 46 Version 2 participants, and 94 (20.9%) out of 367 Version 3 participants responded to the survey request. For follow-up 3, 17 (26.6%) out of 64 Version 1 participants, 2 (3.9%) out of 51 Version 2 participants, and 83 (18.8%) out of 441 Version 3 participants responded to the survey request.

Discussion

This study describes the development of, and improvements made to a smoking cessation web app developed in Spanish and English. The first two versions of the web app were designed by the research team and programmed by an external team. The third version was programmed and designed by a member of the research team. The changes made between Version 1 and 2 were focused on increasing functionality and adding interactive tools. The development of Version 3 was focused on improving visual design and user experience, with only minimal changes to content, to examine the impact of site design on user engagement.

Results indicated that engagement improved substantially in subsequent versions of the web app, which was likely driven mostly by design changes rather than changes in features or content. While text messages seem to be a preferred method of communication for most individuals, implementing them remains a challenge because many users did not provide a valid number or requested that text messages be stopped. Overall completion rates of follow-up surveys were low. In order to understand the effectiveness of the interventions, follow ups are still necessary, but the ability to follow up with participants continues to be an ongoing and growing challenge for digital interventions, even when participants are compensated. The field needs to develop creative alternatives to increase follow-up rates. In addition to finding appropriate participant incentives, other options may include the utilization of unobtrusive tracking (through passive data collected by mobile phones or wearables), or push notifications through native apps.

A set of lessons learned from the development process and usability outcomes of a web app for smoking cessation were identified. First, implement small iterative studies. Developing and testing several versions allow researchers to gather feedback from users and objective data to guide the subsequent versions and improve their product. While this may be the norm in the tech industry, the research and academic fields tend to work at a slower pace, making these iterations more challenging. Although it may require additional efforts to do rapid iterations in the academic field, the final product will likely be more successful and engaging.

Second, utilize a broad number of data sources to inform future development. In our process, we used information from interviews, data from the previous versions of the web app, and knowledge about user experience and visual design. The interviews with low-income smokers demonstrated low mobile literacy and lack of previous experience with smoking cessation apps. Together with utilization information gathered from earlier versions, this prompted us to greatly simplify the app and to focus on ease of use for the final version.

Third, ease of communication with the developer is key. The first two versions of the app were programmed by an outside professional development group whereas the final version was developed by an in-house developer knowledgeable about the subject matter and invested in the success of the project. These factors greatly facilitated continual productive interactions between research team with the developer, resulting in greater responsiveness and ultimately, rapid improvements in the web app.

Fourth, for modern applications, design and appearance matters. This may sound obvious for those in the tech industry, but researchers are generally not trained in design or user experience and may not emphasize these aspects. The final version of the web app had a vastly different look and feel from the previous two (i.e., more simple color scheme, less text and fewer clicks needed, sleek and modern, responsive sizing, improved landing page with feature descriptions), which resulted in better engagement and higher utilization.

Fifth, identify appropriate recruitment channels. While this is not strictly a lesson learned from the development process, finding proper recruitment channels can influence the type and amount of feedback needed for the development of subsequent versions of the intervention. One of the intentions of the study was to develop succeeding versions of a smoking cessation web app to increase engagement of Latinx individuals, with a special focus on Spanish speaking smokers. The recruitment of a substantial portion of Spanish-speaking smokers for Version 3 was successful: 46% of our consented participants were Spanish speaking. The improvement in recruitment for Version 3 was primarily attributable to our use of social media methods, including Facebook ads (8), suggesting that learning the communication channels used by the population of interest is essential to reaching them.

The development of online interventions has great potential to reach large numbers of smokers, including smokers who would not access other smoking cessation resources (9) or smokers who live in places where smoking cessation services are not provided in their language. We have advocated for the creation of Digital Apothecaries (10) that is, online portals that would serve as repositories for Massive Open Online Interventions (MOOIs) in many languages (11). These “MOOIs” would provide evidence-based health interventions, including those designed to help individuals quit smoking. We encourage the field to create, evaluate, and disseminate such tools.

Conclusions

Online interventions are helpful, but there is still a lot to learn about how to develop them. We described the development and usability of a Spanish/English Smoking Cessation Website in order to share the lessons learned with the research community. In essence, researchers of online and digital interventions could benefit of incorporating concepts from the technological industry such as design, user feedback, user experience, small iterative studies, and most importantly, close work with the development team. While we improved recruitment and engagement across subsequent versions of the web app, there were still challenges in the utilization of text messages and follow-up completion rates. Sharing these experiences within the research community should allow for a faster development of new and more efficacious interventions.

Acknowledgments

Center for Behavioral Intervention Technologies at Northwestern University Feinberg School of Medicine, Jodi Prochaska, Adrian Aguilera.

Funding: This research was supported by funds from the California Tobacco-Related Disease Research Grants Program Office of the University of California (PI: Muñoz), grant number 24RT-0027. This research was supported in part by a Google Ads grant.

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/mhealth-19-255). MHB, RFM, AMC and SR state that one grant was disclosed on several COI forms which came from the California Tobacco-Related Disease Research Grants Program Office of the University of California (PI: Muñoz), grant number 24RT-0027. This grant funded the research activities as well as payment for MHB, RFM, AC, and SR. RFM’s funding through the grant was for his work as the content developer of the web application. Melissa Bond designed the final version of the web application. AC and SR served as research assistants on this grant. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). This study was approved by the Palo Alto University Institutional Review Board (15-042) as well as the University of California, San Francisco Committee on Human Research (15-17597). All participants completed an online consent form prior to engaging with any version of the web app.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Do HP, Tran BX, Le Pham Q, et al. Which eHealth interventions are most effective for smoking cessation? A systematic review. Patient Prefer Adherence 2018;12:2065-84. [Crossref] [PubMed]
2. McCrabb S, Baker AL, Attia J, et al. Internet-Based Programs Incorporating Behavior Change Techniques Are Associated With Increased Smoking Cessation in the General Population: A Systematic Review and Meta-analysis. Ann Behav Med 2019;53:180-95. [Crossref] [PubMed]
3. Muñoz RF, Aguilera A, Schueller SM, et al. From online randomized controlled trials to participant preference studies: morphing the San Francisco Stop Smoking site into a worldwide smoking cessation resource. J Med Internet Res 2012;14:e64 [Crossref] [PubMed]
4. Muñoz RF, Barrera AZ, Delucchi K, et al. International Spanish/English Internet smoking cessation trial yields 20% abstinence rates at 1 year. Nicotine Tob Res 2009;11:1025-34. [Crossref] [PubMed]
5. Bravin JI, Bunge EL, Evare B, et al. Socioeconomic predictors of smoking cessation in a worldwide online smoking cessation trial. Internet Interv 2015;2:410-8. [Crossref]
6. Pérez-Stable EJ, Sabogal F, Marín G, et al. Evaluation of "Guia para Dejar de Fumar," a self-help guide in Spanish to quit smoking. Public Health Rep 1991;106:564-70. [PubMed]
7. Muñoz RF, Marín BV, Posner SF, et al. Mood management mail intervention increases abstinence rates for Spanish-speaking Latino smokers. Am J Community Psychol 1997;25:325-43. [Crossref] [PubMed]
8. Bunge EL, Taylor LA, Bond M, et al. Facebook for recruiting Spanish- and English-speaking smokers. Internet Interv 2019;17:100238 [Crossref] [PubMed]
9. Muñoz RF, Chen K, Bunge EL, et al. Reaching Spanish-speaking smokers online: a 10-year worldwide research program. Rev Panam Salud Publica 2014;35:407-14. [PubMed]
10. Muñoz RF, Chavira DA, Himle JA, et al. Digital apothecaries: a vision for making health care interventions accessible worldwide. Mhealth 2018;4:18. [Crossref] [PubMed]
11. Muñoz RF, Bunge EL, Chen K, et al. Massive open online interventions: A novel model for delivering behavioral-health services worldwide. Clin Psychol Sci 2016;4:194-205. [Crossref]