Refining a digital phenotyping app for measurement of suicidal behavior among minoritized youth and caregivers in a community health system

Introduction

Background

Driven in part by increases in suicidal behavior among youth, several national pediatric healthcare organizations have declared a national emergency in child and adolescent mental health (1). Suicidal behavior persists as a public health emergency that has increasingly affected youth from racial and ethnically minoritized groups (2,3). Trends in suicide attempt and injuries over the past decade have remained higher for Black and Latinx youth as compared to White youth, and the coronavirus disease 2019 (COVID-19) pandemic appears to have exacerbated suicide risk and death among Black children (4-6). Elevated rates may be related to adverse social experiences, such as discrimination, family conflict, and trauma (5). Such adverse social experiences align with a developmental application of the influential interpersonal theory of suicide, which explains that thwarted belongingness is particularly salient for youth at risk for suicide (7). New approaches to measuring such experiences may help explain and reduce the persistent disparities in continuity of mental health services among these groups of suicidal adolescents (8).

Digital phenotyping as a promising tool to assess and predict suicidal behavior

Digital phenotyping is defined as “moment-by-moment quantification of the individual-level human phenotype in-situ using data from smartphones and other personal digital devices” (8). This approach is a novel method for obtaining passive data from smartphones to better understand the context (e.g., geospatial location, phone use, etc.) of unsafe thoughts and behaviors. Indeed, digital phenotyping has already been used in the study of suicidal behavior among youth (9-11). This early work has shown that measuring suicidal behavior via digital tools is a safe, feasible, and acceptable approach to youth and, notably, it does not worsen such behaviors (12). This method can also be used to measure risk factors for suicidality that are particularly relevant to minoritized individuals, such as experiences of discrimination (9).

Suicidal behavior is an interpersonal phenomenon (13), where the quality of one’s social relationships and feelings of belonging can have a major impact on self-injurious behaviors and motivations. There may be important insights to learn from the interactions between youth and caregivers that can be measured via smartphones (e.g., using push notifications), such as family conflict and mood. Indicators of family interactions can also be accounted for by studying the passive (or opportunistic) cell phone use for each family member, which can include changes in location or number of calls and texts. Therefore, when used to study youth suicidal behavior, digital phenotyping in combination with other forms of data collection directly from users arguably should be extended to include input from caregivers (parents or legal guardians), since family conflict is a known risk factor for suicidal thoughts and behaviors (STBs) (14). To our knowledge, the available literature on smartphone measurement of digital phenotypes among suicidal youth does not include measurement from family members, and the present study addresses this gap.

App development

In this study, our team modified an earlier version of our smartphone app that has been used to study mental health challenges (15) and other health conditions, such as liver disease (16), to facilitate the identification of digital phenotypes that describe STBs. The app collects passive phone data (opportunistic sensing) and active user input (participatory sensing) with the goal of identifying digital phenotypes that describe suicidal behavior. Specifically, the smartphone app was designed to capture privacy-preserving passive data on how youth interact with their smartphones in their daily lives via summaries of accelerometer and gyroscope data, Wireless Local Area Network (WLAN) connectivity, Global Positioning System (GPS) location, and metadata on smartphone communications. These data allow inferences about sleep, physical activity, and other social and phone interactions. The app also collects short-response survey data through ecological momentary assessment (EMA) via brief questions about daily mood, suicidal and self-injurious behavior and thoughts, family conflict, and discrimination. Because the app will measure information about social context and emotional well-being, we have provisionally named the app “GeoMood”. For safety and ethical reasons, the app also includes an evidence-based suicide safety plan so that a participant would have easy access to a list of coping strategies and safety contacts (17).

User-centered and codesign approaches to mobile health technology (mHealth) are widely recommended to align app functions with user needs and preferences (18,19). GeoMood was developed using a hybrid development approach that combined principles of user-centered and codesign frameworks. The development phase of GeoMood, which resulted in a functional app prototype, involved app developers and subject matter experts on suicidality among minoritized youth (i.e., clinicians and researchers, several of whom are underrepresented in this field of research) who informed the app’s content. This was followed by an evaluative phase to further refine the app through user input (20). This user input included both youth and caregivers. Caregiver input was invited because the family context is so important for understanding adolescent suicide risk (14) and a caregiver version of the app is envisioned for this work. Stakeholder feedback from both youth and caregivers was therefore deemed necessary to improve the acceptability of the app for adolescents at risk for STBs (19,20) to prepare for a pilot trial. We define acceptability as “how the intended individual recipients—both targeted individuals and those involved in implementing programs—react to the tool” (21). Similar approaches to using stakeholder feedback to refine an app have been documented in the medical literature (18). This study presents findings from the qualitative interviews that were conducted with youth and caregivers during the final refinement stages of the app design process. This manuscript was written following the Standards for Reporting Qualitative Research (SRQR) reporting checklist for qualitative research (22) (available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-24-39/rc).

Methods

Eligibility and recruitment

Adolescents with a past-year history of suicidal ideation or self-injurious behaviors and their respective caregivers were recruited to provide insight on their views about the GeoMood app. Eligible youth were identified via review of electronic health records (EHR) while receiving inpatient care in an adolescent psychiatric unit in a safety-net health system serving culturally diverse, lower income patients in metro-north Boston. Eligible youth participants were adolescents and young adults (12–24 years old) receiving outpatient mental health treatment at time of recruitment; who spoke Spanish or English; and who endorsed a past-year history of suicidal ideation or self-injurious behaviors. This relatively broad age range is supported by prior research that has validated the suicide assessment items selected for the app in this age group (23-25). Participants under the age of 18 years were required to have consent from a caregiver. We diversified the sample by selectively recruiting racial/ethnic minoritized youth (as identified via clinician notes in the EHR) who were recently admitted to the psychiatric inpatient service. Caregivers were also invited to be interviewed if they met the inclusion requirement of being English- or Spanish-speaking. The Institutional Review Board (IRB) of Cambridge Health Alliance (CHA) approved this study (No. CHA-IRB-1161/09/20), and the study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). All participants gave their informed consent (caregivers) and assent (youth) before participating in this study and were reimbursed for their participation with a $10 gift card.

The GeoMood app

As noted above, GeoMood is an mHealth app, designed and engineered by authors (T.G.R. and E.S.), that collects passive data, such as activity level (GPS data, Wi-Fi connectivity based on motion and location), social interaction (text and call frequency and duration), and phone motion and interaction (screen awakenings and app switching). The app also serves as an EMA tool that sends time-stamped survey prompts to participants at scheduled intervals. This notifies the user to complete questions in the app related to mood and physical state (via an emoji selection), STBs, family conflict, and experiences of discrimination or exclusion. The Everyday Discrimination Scale also included in the app. This is a widely used measure of discrimination and to our knowledge, this is the first time this measure has been incorporated into a mobile app (26). Survey measures hosted on the app with their related evidence base are described in Table 1. The app runs on both the iOS and Android operating systems and is available in two separate versions: one tailored for youth and another customized for caregivers. The version for caregivers is similar to the youth version except that it does not ask questions about STBs. The updated version of the app is undergoing testing and has not yet been deployed to the intended audience.

Qualitative interviews

Qualitative semi-structured interviews were conducted by the first authors to obtain detailed input from youth and caregivers, the intended users of GeoMood, on the app’s core components: collection of passive data, reports of suicidal and self-injurious behavior, family conflict, experiences of discrimination, and the app’s safety plan. The goal of this data collection approach was to make this app acceptable to future users.

An interview guide was developed by the research team, which includes members with expertise on suicidal behavior, digital technology, and qualitative research methods. The development of the interview guide involved multiple drafts and revisions conducted by the research team. The interview guide included questions organized under six domains: (I) use of the app to track experiences that could be connected to STBs; (II) app appearance; (III) app acceptability and content; (IV) use of the app for safety planning; (V) experiences of discrimination/exclusion, family conflict, and their relevance to suicidality; and (VI) frequency of surveys (Appendix 1). This guide was used to conduct 60-minute, semi-structured, in-depth virtual interviews using a Health Insurance Portability and Accountability Act (HIPAA) compliant video conferencing platform. Interviews were recorded for transcription and coding purposes. The goals of the interview were to gauge participants’ views of the app’s acceptability to (I) collect passive data from their smartphones (opportunistic sensing); (II) assess STBs and related risk factors (i.e., family conflict and discrimination/exclusion); and (III) to obtain input on app modifications that had the potential of increasing user engagement.

Data collection

The interview process involved presenting screenshots of the app’s features and asking questions from the interview guide. The interview began with a general description of the app, which included a summary of its purpose and features. After that summary, participants were prompted to share their immediate impressions about the app and were asked how they felt about the app’s intended collection of passive data. Participants were then presented one screenshot at a time and asked to share their input regarding appearance, layout, acceptability, and content. After presenting the app’s features and gathering input, interviewees were asked questions about experiences of discrimination, exclusion, family conflict, and their views about how these experiences were relevant to suicidality.

Reliability (i.e., consistency and accuracy) of the qualitative interviews was achieved by having the same two-interviewer team conducting the interviews. This approach introduced consistency across interview sessions. With regards to the validity of the qualitative interviews, an examination of the questions included in the interview guide indicates alignment with the goal of capturing input from interviewees on the app’s appearance, layout, acceptability, and content.

Data analysis

Verbatim transcripts of the interviews were uploaded to Dedoose, a qualitative data analysis software program (29), for data coding purposes and analyzed using a qualitative manifest content analytic approach (30). This strategy was selected because the type and intent of the gathered data did not revolve around uncovering or analyzing inferential evidence. Instead, the focus was on maintaining proximity to the text, that is, describing what research participants responded to questions asked, which did not involve identifying underlying mechanisms shaping participants’ responses (30). Following this manifest content analytic approach, the coding strategy involved generating an a priori codebook guided by the interview guide’s domains (Appendix 2). This deductive approach to coding was chosen to ensure that the study’s goals were aligned with designing an app that reflected the preferences and needs of potential future users. The interviews were coded by the research team’s senior qualitative researcher. The decision to employ a single-coder method stemmed from the precise formulation of interview questions, which aimed to capture participants’ responses and evaluations directly related to the app’s features. This is very different from other kinds of qualitative interviews such as phenomenological interviews that aim to elicit stories about lived experiences to gain deeper insights of a particular event or experience. The questions asked during the interviews conducted in this study were designed to capture specific reactions and preferences to the app’s features from potential future users.

Results

Participant characteristics

A total of 10 participants were recruited for this study (five youth and five caregivers) (31). Both the youth and caregiver samples were predominantly female (n=4, per subsample). Results from the data analysis are presented below under the six content domains included in the interview guide. Youth participants are labeled as Y01 through Y05 and caregivers are labeled as C01 through C05.

Domain 1: use of the app to measure STBs

Participants were asked to share their impressions after being presented with a summary of the mobile app’s features. The suicide-related questions on the app ask about both the intent and the desire “to kill yourself right now” (24). Considering the nuanced difference between “intent” and “desire”, participants were asked to share how they understood the two questions. The reaction to these concepts differed between youth and caregivers.

Caregivers were generally more concerned than youth participants about the items being repetitive.

• C01: “I mean, I know you are trying to get to a specific idea, but because the first two questions, it’s desire and intent, I understand that they’re different, but to a kid, they’re going to look like they’re going to be the same and they’re going to go why am I answering the same question more than once?”

A caregiver also expressed concern about the app triggering self-injurious thoughts by asking youth about their suicidal desire and intent twice daily.

• C02: “I don’t know, it’s just my perception as a mom, but I feel like it is always reminding them that they can harm themselves at any moment.”

Youth had a different perspective on asking about both desire and intent.

• Y01: “I like that you guys did the difference between your desire and your intent because sometimes I really want to kill myself, but I won’t.”

Another participant confirmed the acceptability of direct language.

• Y02: “if you used another word that kind of hop-scotched around the same idea, you might not get the results that you want because it might be up for interpretation.”

Y02 also offered a suggestion to differentiate desire from intent:

“Maybe if you bold the word like you have done with other questions.”

Table 2 further illustrates how caregiver and youth responses diverged on the question of assessing specifics of suicidal thoughts.

Youth participants also gave suggestions for wording of survey items based on their experience of seeing and using suicide-related terms on popular social media:

• Y02: “I know that a common term that is used on social media a lot is ‘unalive yourself,’ instead of ‘kill yourself.’ It’s kind of a softened version of saying the same thing. You will see it used a lot in social media, like in TikTok and Instagram, because ‘kill’ is like a triggering word. ...So if you don’t want to use the word ‘kill yourself’ and you want to use something that younger people are more familiar with and commonly see, you can say the word ‘unalive.’ It might not be as triggering; it might not be as blaring.”

Based on the above input from participants, the words “desire” and “intent” were bolded in the GeoMood app to emphasize their difference for users. Otherwise the language of the suicide-related items were kept as is to maintain fidelity to examples from the research literature (24).

Participants were also asked to provide input about two other questions assessing suicide plans and occurrences of self-injury: (I) “Are you right now or were you recently thinking about attempting suicide or hurting yourself to die?” (delivered at a random time during the day); and (II) “Did you do anything to hurt yourself with or without wanting to die today?” (delivered at the end of the day). Youth and caregivers contrasted the wording of these questions in relation to the questions that ask about current intent and desire “to kill yourself right now”, and a question about the ability to stay safe.

One youth commented on the formal or high-register wording of the questions:

• Y02: “These questions are a bit more formal I suppose, they don’t use the term ‘kill yourself.’ They are a bit more detailed because they’re longer, they’re more specific.”

One other youth, commented on the importance of including both types of questions:

• Y05: “They are different questions and I think, generally, should ask like both, I think.”

A caregiver offered a suggestion on the standardness of the questions.

• C03: “...the first set of questions, you’re saying, like, do you want to kill yourself, and then this one says suicide. Like, I think that it should be standard throughout, like I think it should be worded the same way throughout.”

Youth and caregivers also offered suggestions to separate suicide and non-suicidal self-injury to get a more specific answer to the “right now were you thinking” question at a random time during the day.

• Y01: “I don’t know, you could probably add this, but like right now are you thinking about hurting yourself instead of just suicide, because you might just want to hurt yourself.”

A caregiver suggested the question about suicide only appear if the youth responded affirmatively to the question about non-suicidal self-injury.

• C03: “…first they could think about hurting themselves, and from there the next step could be killing themselves, but I think that it’s a pretty direct and raw question. I will say again, maybe start with have you thought about hurting yourself, and depending on that answer, the next one will appear, that is my opinion. Ask the question to those who are giving those indications that they could [attempt], but to the others, no.”

The above responses illustrate the concerns caregivers expressed about using direct language to assess suicidality among youth. Because such items have been used safely in an acceptable way in prior research, our team decided to maintain the same language in GeoMood (24).

Domain 2: app appearance

Participants provided feedback on GeoMood’s appearance. The first feature that users see when they open the app is an emoji-based mood tracking tool with the query, “How do you feel today?” (Figure 1). This feature was created in order to provide a very simple and friendly interface that quickly facilitates entry of critical data points on a user’s mood and physical state, that can also provide contextual information for EMA responses as well as for the collected passive data. Users answer this question by selecting one “emoji” (a small digital image of a cartoon face expressing an emotion or physical state) that best reflects how they are feeling at the moment. GeoMood is programmed to prompt users to respond to this question twice a day. While only prompted twice per day, users are able to visit this page and make entries as often as they like.

Figure 1 Emoji mood items.

Overall, both youth and caregivers approved of using emojis to collect information on mood and physical state.

• C01: “I think they [youth] use their phone a lot and they’re always texting and using emojis and shortcuts. I think that works for your target.”

The use of non-verbal images to capture the current emotional state was appealing.

• C01: “I think that youth would have an easier time using an emoji than trying to explain how they feel, so I think that is a good point.”

Participants in the youth group had mixed thoughts regarding the emojis’ appearance.

• Y01: “I like them, they’re cute. The ears are a little weird, but they’re cute.”

Other youth respondents did not like the emojis’ appearance. Although, when prompted, the app shows a short label describing the emotional state that the emoji reflects, one participant had concerns about the clarity of the emojis while others were concerned about the quality of the emoji appearance.

• Y03: “Some of these are a little confusing”; “I’m not staring at this ugly thing every single day.”
• Y02: “I would say that the emojis are rather underdeveloped relative [to] what kids usually see on social media and on their keypads.”

Domain 3: app acceptability and content

Participants provided feedback on the acceptability of the app, or how they reacted to the presentation of the app’s features in terms of satisfaction and perceived appropriateness. There initially appeared to be a misunderstanding of the app’s purpose, or at least a preference for an app that worked as an intervention to keep youth with STBs safe. We clarified that the app was designed as a measurement tool, although it does have a safety plan function. A caregiver suggested the app should be able to differentiate what kind of help a respondent might need based on the severity of their answers:

• C04: “I think that would be a good idea to differentiate does this person need help right now versus can a physician call or is it best if they can, you know, get a call from someone, you know, at the end of the day or do they need help right now.”

Participants had questions regarding their acceptance of passive data collection by the app and related concerns about their privacy. When given more detailed information about the type of passive data collected (location, phone awakenings, text messages sent), all participants indicated they were comfortable with the relevant privacy implications. A youth explained why these passive data were, in their view, acceptable to gather:

• Y04: “I think that it captures, you know, like those parts that it needs to and it’s not capturing things that are not necessary and it’s also like, you know, you get privacy while having the app, so I think it captures the good things, what it needs to capture.”

A caregiver shared a similar sentiment about acceptance of the app’s function of collecting private information:

• C04: “...for a kid, I think as long as the parent is involved and they know, you know, I mean our phone is tracked in so many other ways that we don’t know about, you know? If it’s in the best interest of the child, I think it is okay.”

Domain 4: use of the app for safety planning

During the interviews, participants reviewed screenshots of the app’s safety plan section. Both youth and caregivers seemed familiar with this function of the app from prior experience with safety planning in mental health treatment. The app’s safety plan section includes elements of the Stanley-Brown safety plan, such as activities to distract from unsafe thoughts, as well as contact info for support from preferred individuals and professionals (32). Overall, both caregivers and youth participants reacted favorably to the safety plan. Caregivers appreciated that youth had access to important resources that users may need in the moment. However, some youth questioned the role of “distractions” as written in the safety plan (specifically, the plan template asks for “People and social settings that provide distraction”):

• Y01: “...sometimes like you don’t need distractions, you just need someone to be there with you.”
• Y03: “it’s another word [for] distraction, you could use comfort. I think that that wording [distraction] needs to change.”

Youth participants recommended the app allow updates to the safety plan over time.

• Y01: “Because what if you have another warning sign you realize or another person you have?”

A participant proposed an innovation on the use of contacts other than email and phone numbers:

• Y03: “what if the person has their phone taken away, then you just message them on Instagram or what if the person has, like, an Android, you can just message them on Discord [social media].”

Finally, caregivers recommended having a prominent tab or pop-up within the app for easy access to the safety plan. The design of the safety plan in GeoMood does allow for easy access to the safety plan (immediately following the day’s survey items), and its content is designed to be revisable by the user. Since the Stanley-Brown safety plan has been studied with adolescents, we decided to keep the original wording, knowing that a research team member will assist the participant to complete the safety plan section of the app at the beginning of the study.

Domain 5: experiences of family conflict, discrimination/exclusion, and their relevance to suicidality

Participants were asked about using the app to assess family conflict and discrimination or exclusion in relation to STB. While some participants preferred these questions be prompted less frequently, participants generally indicated it was relevant to ask about these experiences.

• C05: “Absolutely. It’s really, absolutely, trust me, and in this environment that we’re in right now with everything going on with racial disparity and implicit bias, absolutely 100%.”

Participants saw an opportunity for capturing experiences of bullying via the discrimination-related questions:

• C03: “So, whereas my daughter isn’t, isn’t the subject of bullying, other children her age who are going through similar things might be and… that’s not an easy thing to open up and just be like, hey I’m being bullied, but maybe something as simple as an app and being able to answer questions like that, it might throw up a red flag for somebody else to be like, hey, check out, you know, what’s going on and why they might answer this way.”

Another participant similarly approved the app’s potential to assess a diverse range of characteristics that might lead to discrimination:

• Y03: “... do you receive poor[er] service than other people at restaurants or stores because [of] your race, because [of] your sexuality, because [of] your mental state. ‘Cause I’ve seen autistic kids and neurodivergent kids get treated extremely badly at schools…”

With respect to measuring family conflict, participants recommended more specificity and inclusion of these items. One participant proposed asking about relationships with other relatives as opposed to asking about “family”. A caregiver suggested allowing the app to capture more information about family conflict:

• C04: “I think the language is clear, it’s not too overloaded, the questions, those are good. If there was maybe a box or maybe in the next section maybe like a box where they can write something specific just to have them get out, you know, their words or their expressions about something that happened, I think that would be good.”

Domain 6: frequency of surveys for suicidality and mood

When asked about how frequently the app should prompt users to answer questions about self-injurious thoughts and behaviors, caregivers differed in their viewpoints. One thought it would be beneficial for this question to be prompted more than once a day while another thought it would be best not to prompt this question at all.

• C03: “I think that for it to be asked twice in the same day is okay…. Sometimes [my daughter] will wake up in the morning and she’s fine, but then at nighttime, when she’s in her room and it’s time to go to bed, she said the nighttime is when she has more thoughts than she does throughout any part of the day.”

Youth participants recommended varied frequency depending on behavioral patterns.

• Y03: “you could do a little research on if there is a specific time that the user feels most suicidal; like for some people it might be in the evening cause they’re at home and they’re lonely or something…”
• Y04: “you should also have a follow-up question like ask them have you attempted in the last couple of weeks or couple of months to see, like, if there is a pattern of something happen[ing] every those couple of weeks….”

When asked about how frequently the app should prompt users to report their mood or physical state, both youth and caregivers recommended an evening prompt, especially if the goal was to capture overall daily mood. They also suggested more than one prompt per day.

• Y01: “because in the mornings you don’t really know because your day hasn’t even started yet. And afternoon, there could be, like, a lot that changes, so I would be around right before you go to sleep...what if you fill it out in the morning, or one time, and then you have something that happens, like a bad thing that happens right afterwards?”

One caregiver agreed with checking for mood in the morning and evening.

• C04: “So, maybe shortly after school starts, maybe like 9:00, just to capture maybe the morning and then sometime right before school ends or maybe when school ends… once a day is very little information it seems to me.”

Since the planned timing of suicide assessments covers most of the suggestions above, no further changes were made in their frequency. Building an adaptive component of the app (i.e., tailoring the timing based on individual patterns) is an innovative suggestion that is outside of the scope and resources for the app design at this time. However, this feedback is noteworthy and will be considered when the app is actually deployed.

Discussion

The current study used a codesign approach to refine an mHealth smartphone app focused on the use of participatory (i.e., via improved measurement of STBs and related risk factors of family conflict and discrimination) and opportunistic sensing (i.e., passive data) to inform strategies that could lead to the reduction of adolescent suicide disparities. Feedback from both youth of color and their caregivers was invited to refine the development of two versions (i.e., youth and caregivers) of the app, thereby providing a means to capture family experiences. This approach provided meaningful insights into a tool for digital phenotyping. The data from digital phenotyping can be used to help monitor behavioral and physiological patterns on a more frequent basis, providing a more accurate picture of STBs and related risk factors. By continuously tracking changes in sleep, social interactions, and movement, digital phenotyping enables personalized tools that may help patients and providers anticipate crises before they escalate. The approach may inform future apps addressing youth suicide reduction and/or measurement of relevant risk factors. The approach may inform future apps addressing youth suicide reduction and/or measurement of relevant risk factors.

Key findings

The aim of this qualitative study was to engage five diverse youth and their caregivers in the refinement of an app to be used for measuring STBs, mood and physical state, family conflict, and experiences of discrimination as well as measuring passive phone use. The findings suggest that youth with a past-year history of suicidal behavior found the GeoMood app to be an acceptable and useful tool for its intended purpose of measuring both self-report and passive phone use data, while caregivers expressed concern over the direct phrasing of the suicide-related items. There was strong support for measuring experiences of discrimination and exclusion via the app. Our qualitative findings support prior work in this area that found similar tools to be acceptable for measuring STBs in adolescents (9,32). By interviewing both youth and caregivers, we identified how family members viewed the same app somewhat differently. The findings point to a divergence of opinions across caregivers and youth, on how explicit the app should be in measuring suicidal desire, intent, and inability to stay safe. We identified caregiver concerns that asking such questions could trigger a youth to harm themselves. However, youth respondents, all of whom had experience with mental health treatment, both understood and approved of an app that differentiated different kinds of self-injurious thoughts and behaviors. This supports the acceptability of apps designed to measure such sensitive psychological experiences. The use of emojis to assess mood elicited mixed responses from participants. Caregivers saw value in using non-verbal responses to capture mood, but youth described a higher bar for what they consider appropriate for app-based graphics. They connected their high expectations to the sophistication of social media apps they already use on a daily basis, which highlights the need for the graphics of this app used in the earliest iterations of the tool to be updated in our current version in order to reflect present design trends.

Overall, input from both caregivers and youth proved valuable during the app’s refinement process. The study team took participants’ suggestions into consideration and balanced the incorporation of these ideas with the integrity of the app and the evidence base for using each measure.

Comparisons with other research

The use of smartphones to measure patterns of and better understand suicidal behavior among youth is an active area of research. Prior work has demonstrated that using an app for frequent assessment of mood and suicidal ideation is a feasible method of recognizing variability in these outcomes (9,11). From such studies, there is evidence that there are several different patterns of suicidal ideation that can be distinguished by their mean scores and variability, and that participants with higher sustained suicidal ideation are more likely to have recent suicide attempt (10). Some prior research confirms that adolescents and caregivers seem open to the idea of mobile sensing and digital phenotyping. Our study, however, revealed some hesitation among caregivers (33). Similar hesitancy was found in an EMA study of Black men, which found that respondents were less likely to complete suicide measures when experiencing negative moods. This study also made efforts to record race-based stress via voice memo functionality, whereas GeoMood intends on using the more structured Everyday Discrimination Scale.

Moreover, there is also a growing literature on the use of safety planning interventions in mHealth applications. A recent systematic review of such safety planning apps suggests that most were designed, like GeoMood, to be revisable and link to community support (34). Users saw such safety apps as most relevant during times of crisis. Our participants suggested further innovations for revising a safety plan (e.g., including social media contacts in addition to phone numbers). This review suggested a need for involving family members in safety planning, which could be met through the caregiver version of GeoMood.

Implications

This research serves as a promising example of an acceptable precision prevention app that targets a public health challenge in adolescent mental health. Our results have important implications for improving mental health, standard care, and patient engagement among minoritized youth. The GeoMood app holds potential for improving suicide research among minoritized youth, a group that has displayed increasing rates of suicidality (4-6). To our knowledge, GeoMood is among the first to integrate assessments of discrimination and exclusion (via the Everyday Discrimination Scale) into a digital phenotyping app. Data derived from this app can reveal the relationships over time between such stressors and STBs. The passive phone use data captured by the app can be correlated with self-reports to make ever more fine-grained measurements of where and when such experiences occur. An innovative future research direction of a tool like GeoMood is to link phone data to large geospatial datasets that may, for example, reveal links between suicidal behavior, self-reported experiences, phone GPS data, and larger environmental and neighborhood factors (e.g., zip code level rates of socioeconomic inequality or violence) or pervasive social disruptions (e.g., the COVID pandemic) (35). Black youth appear to be especially amenable to sharing health information through technological tools (some research reports 34%, versus 20% of White youth), showing promise for these apps to be a potential tool to address increasing rates of STBs in minoritized groups (36).

In terms of future clinical applications, apps like GeoMood may ultimately find relevance in adolescent psychiatric services. With further study and refinement, including EHR integration, GeoMood may be used in outpatient care as a way to assess risk information and respond to suicidal behavior. Participatory sensing via EMA and opportunistic sensing can allow clinicians to holistically examine adolescent behavior, offering more information during assessment of adolescent suicidality. Although these technologies can be intrusive and may pose privacy concerns, a nuanced ethical approach can help app developers utilize safe and acceptable measures for clinical settings (37).

The codesign process used in this study has the potential to improve patient engagement. Studies have shown increasing patient engagement improves patient wellbeing, health outcomes, and quality of care (38). As such, participatory methods could be a crucial factor in enhancing mental health care, and alleviating poor mood and STBs (39-41). Future apps designed to alleviate STBs may need to consider including a codesign process.

Strengths and limitations

Strengths of the current research include the focus on youth from minoritized groups and the engagement of caregivers in the codesign process. The perspectives of both youth and caregivers are under-represented in the suicide literature (33). Centering the codesign work around families of color can improve equitable access to research tools for measuring risk factors, like discrimination, that merit further investigation (42). Although the small sample included in this study may be viewed as a limitation, the use of qualitative semi-structured interviews allowed for the collection of detailed input from participants regarding the app. Results from this qualitative research played a crucial role in refining the methods and identifying key challenges in the pilot, before scaling up to a larger study. The detailed feedback from this small sample of participants ensured that the app will be better tailored to the target population (43).

Conclusions

Incorporating feedback from both caregivers and youth was an effective way to tailor GeoMood to user needs. Although there were some differing opinions between youth and caregivers on some aspects of the GeoMood app, such as whether or not to include direct questions on suicidal ideation, there was consensus on the acceptability of the app as a tool to assess STBs. There was also consensus on including questions related to discrimination as a suicide risk factor, providing insight into the relevance of GeoMood to minoritized populations. In addition, feedback suggested multimodal assessments (text and emoji prompts) should be incorporated into future mHealth apps.

Integration of apps like GeoMood into standard psychiatric care could improve assessment of suicidality. Future research is needed to understand the clinical relevance of EMA apps (that also passively collect behavioral data) and how to best implement them in healthcare settings using an approach that protects patient privacy and safety while serving as a supportive outpatient tool.

Acknowledgments

None.

Footnote

Reporting Checklist: The authors have completed the SRQR reporting checklist. Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-24-39/rc

Data Sharing Statement: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-24-39/dss

Peer Review File: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-24-39/prf

Funding: This work was supported by the Foundations of Human Behavior Initiative (FHB) and the Star-Friedman Challenge for Promising Scientific Research, both from Harvard University and by the National Institute of Mental Health (No. P50MH126283).

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-24-39/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.The IRB of Cambridge Health Alliance approved this study (No. CHA-IRB-1161/09/20). The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). All participants gave their informed consent (caregivers) and assent (youth) before participating in this study.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. AAP-AACAP-CHA Declaration of a National Emergency in Child and Adolescent Mental Health [Internet]. [cited 2024 Apr 17]. Available online: https://www.aap.org/en/advocacy/child-and-adolescent-healthy-mental-development/aap-aacap-cha-declaration-of-a-national-emergency-in-child-and-adolescent-mental-health/
2. Curtin SC, Heron M, Miniño AM, et al. Recent Increases in Injury Mortality Among Children and Adolescents Aged 10-19 Years in the United States: 1999-2016. Natl Vital Stat Rep 2018;67:1-16. [PubMed]
3. The National Institute of Mental Health. Suicide: Suicide Is a Leading Cause of Death in the United States. [Internet]. 2018 [cited 2023 Nov 11]. Available online: https://www.nimh.nih.gov/health/statistics/suicide
4. Bridge JA, Ruch DA, Sheftall AH, et al. Youth Suicide During the First Year of the COVID-19 Pandemic. Pediatrics 2023;151:e2022058375. [Crossref] [PubMed]
5. Gomez J, Miranda R, Polanco L. Acculturative stress, perceived discrimination, and vulnerability to suicide attempts among emerging adults. J Youth Adolesc 2011;40:1465-76. [Crossref] [PubMed]
6. Mayne SL, Hannan C, Davis M, et al. COVID-19 and Adolescent Depression and Suicide Risk Screening Outcomes. Pediatrics 2021;148:e2021051507. [Crossref] [PubMed]
7. Van Orden KA, Witte TK, Cukrowicz KC, et al. The interpersonal theory of suicide. Psychol Rev 2010;117:575-600. [Crossref] [PubMed]
8. Freedenthal S. Racial disparities in mental health service use by adolescents who thought about or attempted suicide. Suicide Life Threat Behav 2007;37:22-34. [Crossref] [PubMed]
9. Glenn CR, Franklin JC, Nock MK. Evidence-based psychosocial treatments for self-injurious thoughts and behaviors in youth. J Clin Child Adolesc Psychol 2015;44:1-29. [Crossref] [PubMed]
10. Kleiman EM, Turner BJ, Fedor S, et al. Digital phenotyping of suicidal thoughts. Depress Anxiety 2018;35:601-8. [Crossref] [PubMed]
11. Auerbach RP, Lan R, Galfalvy H, et al. Intensive Longitudinal Assessment of Adolescents to Predict Suicidal Thoughts and Behaviors. J Am Acad Child Adolesc Psychiatry 2023;62:1010-20. [Crossref] [PubMed]
12. Pizzoli SFM, Monzani D, Conti L, et al. Issues and opportunities of digital phenotyping: ecological momentary assessment and behavioral sensing in protecting the young from suicide. Front Psychol 2023;14:1103703. [Crossref] [PubMed]
13. Leifker FR, Leo K, Adamo C, et al. Suicide as an interpersonal phenomenon: Dyadic methodological and statistical considerations in suicide research. Suicide Life Threat Behav 2021;51:8-18. [PubMed]
14. Diamond G, Kodish T, Ewing ESK, et al. Family processes: Risk, protective and treatment factors for youth at risk for suicide. Aggress Violent Behav 2022;64:101586. [PubMed]
15. Van Dam L, Rietstra S, Van der Drift E, et al. Can an Emoji a Day Keep the Doctor Away? An Explorative Mixed-Methods Feasibility Study to Develop a Self-Help App for Youth With Mental Health Problems. Front Psychiatry 2019;10:593. [PubMed]
16. Sack J, Reid T, Schlossberg E, et al. A Smartphone App for Patients With End-Stage Liver Disease Can Detect Behavioral Changes That Predict Liver-Related Events. Iproc 2019;5:e15229.
17. Stanley B, Brown GK. Safety planning intervention: A brief intervention to mitigate suicide risk. Cogn Behav Pract 2012;19:256-64.
18. Wray TB, Kahler CW, Simpanen EM, et al. User-centered, interaction design research approaches to inform the development of health risk behavior intervention technologies. Internet Interv 2019;15:1-9. [PubMed]
19. Sanders EBN, Stappers PJ. Co-creation and the new landscapes of design. CoDesign 2008;4:5-18.
20. Sanders EBN, Stappers PJ. Probes, toolkits and prototypes: three approaches to making in codesigning. CoDesign 2014;10:5-14.
21. Bowen DJ, Kreuter M, Spring B, et al. How we design feasibility studies. Am J Prev Med 2009;36:452-7. [PubMed]
22. O’Brien BC, Harris IB, Beckman TJ, et al. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med 2014;89:1245-51. [Crossref] [PubMed]
23. Krall HR, Ruork AK, Rizvi SL, Kleiman EM. Hopelessness as a Mechanism of the Relationship between Physical Pain and Thoughts of Suicide: Results from Two Smartphone-Based Real-Time Monitoring Samples. Cogn Ther Res [Internet]. 2024 [cited 2024 Oct 11]; Available online: https://www.researchwithrutgers.com/en/publications/hopelessness-as-a-mechanism-of-the-relationship-between-physical-
24. Glenn CR, Kleiman EM, Kearns JC, et al. Feasibility and Acceptability of Ecological Momentary Assessment with High-Risk Suicidal Adolescents Following Acute Psychiatric Care. J Clin Child Adolesc Psychol 2022;51:32-48. [PubMed]
25. Mandel AA, Revzina O, Hunt S, et al. Ecological momentary assessments of cognitive dysfunction and passive suicidal ideation among college students. Behav Res Ther 2024;180:104602. [Crossref] [PubMed]
26. Peek ME, Nunez-Smith M, Drum M, et al. Adapting the everyday discrimination scale to medical settings: reliability and validity testing in a sample of African American patients. Ethn Dis 2011;21:502-9. [PubMed]
27. Mansfield AK, Keitner GI, Sheeran T. The Brief Assessment of Family Functioning Scale (BAFFS): a three-item version of the General Functioning Scale of the Family Assessment Device. Psychother Res 2019;29:824-31. [Crossref] [PubMed]
28. Clark R, Coleman AP, Novak JD. Brief report: Initial psychometric properties of the everyday discrimination scale in black adolescents. J Adolesc 2004;27:363-8. [Crossref] [PubMed]
29. Los Angeles, CA: SocioCultural Research Consultants, LLC. Dedoose Version 9.0.17, cloud application for managing, analyzing, and presenting qualitative and mixed method research data [Internet]. 2021 [cited 2023 Nov 11]. Available online: https://www.dedoose.com/
30. Kleinheksel AJ, Rockich-Winston N, Tawfik H, et al. Demystifying Content Analysis. Am J Pharm Educ 2020;84:7113. [Crossref] [PubMed]
31. Bosworth KT, Flowers L, Proffitt R, et al. Mixed-methods study of development and design needs for CommitFit, an adolescent mHealth App. mHealth 2023;9:22. [Crossref] [PubMed]
32. Stanley B, Brown G, Brent DA, et al. Cognitive-behavioral therapy for suicide prevention (CBT-SP): treatment model, feasibility, and acceptability. J Am Acad Child Adolesc Psychiatry 2009;48:1005-13. [Crossref] [PubMed]
33. Orr M, MacLeod L, Bagnell A, et al. The comfort of adolescent patients and their parents with mobile sensing and digital phenotyping. Comput Hum Behav 2023;140:107603. [Crossref]
34. Gryglewicz K, Orr VL, McNeil MJ, et al. Translating Suicide Safety Planning Components Into the Design of mHealth App Features: Systematic Review. JMIR Ment Health 2024;11:e52763. [Crossref] [PubMed]
35. Hinds J, Brown O, Smith LGE, et al. Integrating Insights About Human Movement Patterns From Digital Data Into Psychological Science. Curr Dir Psychol Sci 2022;31:88-95. [Crossref]
36. Well Being Trust, Rideout V, Fox S. Digital Health Practices, Social Media Use, and Mental Well-Being Among Teens and Young Adults in the U.S. Artic Abstr Rep [Internet]. 2018 Jul 1; Available online: https://digitalcommons.providence.org/publications/1093
37. Martinez-Martin N, Greely HT, Cho MK. Ethical Development of Digital Phenotyping Tools for Mental Health Applications: Delphi Study. JMIR Mhealth Uhealth 2021;9:e27343. [Crossref] [PubMed]
38. Silvola S, Restelli U, Bonfanti M, et al. Co-Design as Enabling Factor for Patient-Centred Healthcare: A Bibliometric Literature Review. Clinicoecon Outcomes Res 2023;15:333-47. [Crossref] [PubMed]
39. O’Brien J, Fossey E, Palmer VJ. A scoping review of the use of co-design methods with culturally and linguistically diverse communities to improve or adapt mental health services. Health Soc Care Community 2021;29:1-17. [PubMed]
40. Thomas G, Lynch M, Spencer LH. A Systematic Review to Examine the Evidence in Developing Social Prescribing Interventions That Apply a Co-Productive, Co-Designed Approach to Improve Well-Being Outcomes in a Community Setting. Int J Environ Res Public Health 2021;18:3896. [PubMed]
41. Tindall RM, Ferris M, Townsend M, et al. A first-hand experience of co-design in mental health service design: Opportunities, challenges, and lessons. Int J Ment Health Nurs 2021;30:1693-702. [PubMed]
42. Bath E, Njoroge WFM. Coloring Outside the Lines: Making Black and Brown Lives Matter in the Prevention of Youth Suicide. J Am Acad Child Adolesc Psychiatry 2021;60:17-21. [PubMed]
43. Elfeky A, Treweek S, Hannes K, et al. Using qualitative methods in pilot and feasibility trials to inform recruitment and retention processes in full-scale randomised trials: a qualitative evidence synthesis. BMJ Open 2022;12:e055521. [PubMed]