Refining an ecological momentary assessment study of binge eating among sexual minority and heterosexual young women: a mixed methods pilot study

Introduction

Ecological momentary assessment (EMA) methods are often used to assess real-world behavioral and health-related processes in people’s natural environments. Very broadly, EMA approaches provide a method for measuring features of people’s daily experiences, states, and environments as they are going about their everyday lives. Although EMA can take many forms, collectively these methods share three general characteristics: data are collected in people’s natural environments; events and experiences are assessed close to when they happen; and assessments occur intensively, often from one to several times per day over multiple consecutive days (1,2). EMA studies often, but not always, use mobile technology, such as smartphones or wearable sensors, to collect data more easily from people as they go about their daily lives. As described elsewhere (1), EMA approaches also can include both self-report surveys, which require active participation from people in the study, as well as passive sensing approaches (e.g., wearable accelerometer, heartrate monitor, etc.), which can passively collect data with more minimal participation from study participants.

There are a wide range of design decisions that researchers must make when conducting an EMA study, including selecting a sampling schedule, designing self-report surveys, selecting hardware and software, and developing study-specific materials (e.g., training materials, compensation thresholds and schedules) (2). However, there is often limited prior research identifying optimal study design features, particularly when assessing novel constructs, or when conducting research among new populations (3). For example, for self-report EMA surveys, the ideal survey length, and type and/or number of surveys per day is not always clearly prescribed and depends on a combination of the research questions, constructs being assessed, and sample characteristics. For self-report surveys, the wording of questions and number of items needed to assess constructs is also highly dependent on the individual study and sample; validated EMA measures are far less common than for more traditional survey research designs. Furthermore, when a study uses mobile technology for data collection, attention to the hardware and software selected, as well as the associated training procedures, is important during the study planning. In order to gather information about optimal study design, formative research with the populations of interest can be useful for informing the design and implementation of the study. In the present study, we gathered preliminary mixed methods data on the acceptability and feasibility of EMA procedures with young women ages 18–30, which we used to inform the design of a larger EMA study assessing binge eating among young adult sexual minority (e.g., lesbian, gay, bisexual) and heterosexual women.

EMA design considerations

The timing and frequency of assessments is a critical aspect of EMA study design, and although they should primarily be driven by the research questions being examined, it is also critical that participant burden be considered. Such consideration has been thoughtfully described in a study by Burke and colleagues (4) where they discuss preliminary work conducted in preparation for a 12-month EMA study. In their 21-day pilot study, adult participants reported that a maximum of five prompted surveys per day would be feasible and acceptable without being burdensome. Following this design recommendation, during a 12-month EMA study examining lapses and relapses after weight loss in adults, compliance rates for randomly prompted surveys were at least 85% (4). This study provided evidence that their sampling frequency (i.e., number of prompts per day) and timing (i.e., when prompting began and ended each day) was acceptable to participants and yielded appropriate compliance with the EMA protocol. This study also highlights the role pilot work can play in designing a study that will yield acceptable compliance with the study’s EMA procedures.

In addition to the timing and frequency of assessments, it is also important to consider the content of assessments. Whereas many traditional forms of research rely on existing validated questionnaires to measure various constructs, these are often not feasible for EMA studies due to length and inappropriate time frames for intensive data collection. Thus, researchers must create specific assessment items to use in EMA studies, often adapting items from existing (non-EMA) measures. For example, in recent years researchers have developed and validated EMA measures that can be used to assess suicidal ideation (5), sexual minority stressors (6,7), and social perceptions (8). This work is often conducted in a series of steps or studies, with items first being developed and refined when conducting formative work, prior to being validated. Such work often benefits from piloting newly constructed items with participants to ensure understanding, and several groups have recommended using both qualitative and quantitative methods when designing and refining items (8,9).

Beyond frequency and content of EMA assessments, training is a unique aspect of EMA study design needed to ensure that participants are correctly completing study procedures. Doherty and colleagues (10) highlighted the importance of training participants in EMA studies to encourage study engagement as well as valid data collection. Christensen and colleagues (11) suggested that providing participants with more detail about study questions and opportunities to practice may increase participant compliance. A systematic review of the use of EMA in physical activity and sedentary behavior research noted that some studies provide training in the form of researcher guided assessments, while others implemented test periods where participants pilot the EMA procedures prior to beginning data collection (12). Training materials can be shared with participants in a variety of formats, including during in-person sessions as well as remote sessions either occurring synchronously using video capabilities or asynchronously using recorded or written materials. The optimal training format and method varies based on the sample and other study design features, but there have been some studies evaluating ideal training methods. For example, in a 14-day daily diary study focusing on alcohol use among same-sex female couples, participants were randomized to either receive video and written introductory materials or written materials only (13). Although there were no differences between groups in terms of consent, dropout, or compliance, those participants who spent more time watching videos also had higher rates of survey completion. This type of information acquired during the training period can be used by researchers to screen out participants who may be non-compliant or to identify participants who may need a higher level of support throughout the study (13). As with assessment items, it is important for pilot training materials to assess participant understanding and make data-informed adjustments to training and other study materials (8).

Rationale for the present study

The goal of the present study was to describe a formative mixed methods study used to elicit feedback on EMA study procedures and materials from the target populations for a larger study. Sexual minority women (e.g., lesbian, bisexual, queer) are at greater risk for obesity and binge eating than their heterosexual peers. Previous studies have consistently found that, in comparison to their heterosexual counterparts, sexual minority women are more likely to be obese (14-16). Although studies of binge eating behaviors among sexual minority women are more limited, the existing literature indicates that sexual minority women engage in binge eating at higher rates than heterosexual women (17,18). It is important to understand binge eating in sexual minority women given disordered or unhelpful eating behaviors contribute to obesity (19).

Despite documented disparities between sexual minority and heterosexual women (e.g., obesity rates, binge eating), relatively little is known about factors contributing to these disparities. In the eating disorder literature more broadly, over the last several decades there has been increased attention on assessing disordered eating behaviors in daily life using EMA approaches. Previous studies of general samples of young women (where sexual orientation is not known or reported) suggest binge eating is associated with affective states (20,21), social processes (22,23), and health behaviors (e.g., alcohol use, physical activity) (24,25) in daily life. However, it remains unclear whether associations between these factors and binge eating function similarly for sexual minority women and heterosexual women. Thus, one of the aims of the larger study is to examine associations between affect, social processes, and health behaviors and binge eating in daily life among groups of both sexual minority and heterosexual young women (ages 18–30 years) in an effort to identify similar and/or differential patterns of associations between these groups.

In addition to these potential general predictors of binge eating (i.e., affect, social processes, health behaviors), to date there has been relatively limited research exploring unique factors that may contribute to binge eating in sexual minority women specifically, with two recent exceptions. First, Mason and colleagues (26) conducted a daily diary study with 30 young (18–30 years old), lesbian women reporting binge eating within the past week and examined associations among gender- and sexual orientation-based discrimination, negative affect, self-awareness, and binge eating. Results indicated that experiences of gender and sexual orientation discrimination were indirectly associated with binge eating through increased negative affect and reduced self-awareness. In addition, Panza and colleagues (27) conducted a 5-day EMA study with 55 overweight/obese sexual minority women (18–60 years old) to examine associations between sexual orientation- and gender-based minority stress and disordered eating. These authors found that experiences of sexual minority stressors were associated with more over-eating and binge eating episodes. Although these studies provided some initial evidence for potential associations between minority stressors and binge eating, the impact of minority-specific factors in daily life (e.g., minority stress) on binge eating warrants further attention. Specifically, longer EMA studies with larger sample sizes and direct comparisons to heterosexual women are needed to gain a greater understanding of minority stress and additional factors’ associations with binge eating in sexual minority women. The longer-term goal of this line of research is to use the information gathered from EMA studies to identify potential intervention targets to reduce binge eating for both heterosexual and sexual minority women.

To address these research questions regarding real-world predictors of binge eating in daily life, we are conducting a large EMA study with young sexual minority (target n=150) and heterosexual (target n=150) women ages 18–30 from across the United States. Prior to beginning this work, a pilot study was conducted, which included collecting quantitative data via EMA reflecting the anticipated measures and procedures for the larger study, combined with an end of study survey and qualitative data during an in-depth exit interview to gather opinions on these methods. The aims of this formative pilot study were to gather data on the acceptability and feasibility of the EMA procedures with a specific focus on feedback on: (I) the training and study description materials (e.g., downloading smartphone survey app, instructional videos, etc.); (II) general smartphone app and survey preferences; and (III) EMA survey questions related to binge eating, identity-related stressors, and appearance-related pressures. EMA methods have been used for many years to study binge eating and other related disordered eating behaviors (28). However, it can be challenging for participants to self-identify and record binge episodes in daily life (29,30), and thus, as part of this study we sought feedback on participants’ experiences regarding reporting on binge eating via EMA. In this study, we report the steps taken to refine study materials and procedures based on participant feedback to be implemented in a larger trial. This study is presented in accordance with the STROBE reporting checklist (available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-16/rc).

Methods

Recruitment and participants

A total of 30 participants consented to participate in this pilot study. The eligibility criteria for all participants were: (I) age 18–30 years; (II) identify as a cisgender woman (i.e., were born female and identify as a woman); (III) report binge eating (“overeating and feeling like you lost control of your eating”) at least two times in the past two weeks; (IV) not currently receiving treatment for an eating disorder; (V) have an Apple or Android smartphone; and (VI) be willing to complete brief smartphone surveys for one week and take part in an exit interview. Given we were interested in enrolling sexual minority and heterosexual women, we used a combination of participants reported sexual attraction and identity to select participants. In addition to the above criteria, potential participants were required to report either: (VII) being only or mostly attracted to women and identify as lesbian, gay, or queer; or (VIII) being only or mostly attracted to men and identify as heterosexual or straight.

Recruitment was completed in two phases. First, local recruitment in southeastern Virginia took place in February and March 2020 using flyers and online postings at local businesses and colleges, resulting in 42 potential participants completing the screening survey, of whom 12 were eligible to participate in the study. Nine heterosexual women (and no sexual minority women) consented to participate, and all nine went on to complete all phases of the study, including the demographics questionnaire, smartphone EMA surveys, exit interview, and end of study survey.

Given no sexual minority women were recruited locally, as well as the emergence of the coronavirus disease 2019 (COVID-19) pandemic requiring all procedures to be conducted online, in April 2020 we began a second phase of recruitment, which involved recruiting sexual minority women by emailing potentially eligible women from across the U.S. who had participated in previous cross-sectional survey studies conducted by our research group. These online recruitment efforts resulted in 66 potential participants completing the screening survey and of these 26 were eligible to participate. Of the 26 eligible sexual minority women, 21 consented and completed the demographics questionnaire, and 15 began the smartphone surveys. After completing the smartphone surveys, 10 completed both the exit interviews and end of study survey, 2 were unable to participate in the exit interview (due to scheduling) but did complete the end of study survey, and 3 did not complete either the exit interview or survey. Given that the focus of this paper was reporting on data from the exit interviews and end of study survey, in Table 1 we report demographic information, body mass index (BMI), and number of binge episodes in the past two weeks for the 21 women (9 heterosexual, 12 sexual minority) who completed the exit interview and/or the end of study survey. The sample size was determined when we reached saturation in the interviews.

Procedure

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Intuitional Review Board of Old Dominion University (No. 1493285) and informed consent was obtained from all participants. Prior to providing consent, participants were emailed an introductory video that explained the purpose of the study and smartphone surveys. They also reviewed written materials describing the study and procedures. After providing consent and completing the demographics questionnaire, they were given instructions for downloading the RealLife Exp app from the Apple or Android app store onto their smartphone and for installing the study-specific surveys. Over the following 7 days, participants utilized a smartphone app to complete brief smartphone surveys, which included a user-initiated morning survey to be completed when they woke up and five prompted surveys that occurred at random times throughout the day between 9 am and 9 pm. They were also instructed to complete a user-initiated binge eating survey after experiencing a binge eating episode. Morning surveys assessed sleep quality, alcohol consumption, mood, and upcoming experiences. Prompted surveys included current mood state, body image, stress, eating, health behaviors, and social interactions. Binge surveys assessed reported start and end time of the binge episode and whether participants were with other people during their binge episode. Once participants completed the smartphone surveys for 7 days, they were asked to provide qualitative feedback through a 1-hour interview, followed by an online end of the study survey. We elected to include the end of study survey after the interview as a way for participants to share any additional feedback in a more private, anonymous manner than allowed by a face-to-face interview. Participants earned $50 for completing the week of EMA surveys, and those with an 85% compliance rate on EMA survey completion were awarded a $10 bonus. Also, participants earned $20 for completing the interview and the end of the study survey. Altogether, participants could earn up to $80 for completing all study activities.

Measures

Demographic questionnaire

A demographic questionnaire contained questions to assess age, employment status, educational level, income, financial hardship, ethnicity/race, and sexual identity (select all that apply), and attraction. Participants were asked about the gender(s) of their partner(s) within the past year and their lifetime (e.g., “During the past year, with whom have you had sex?” and “With whom have you had sex in your lifetime?”). Participants were asked to describe their current relationship status. Participants also self-reported their height and weight. Lastly, to assess community resources available to participants, they were asked to provide their address and the type of city or community in which they lived (e.g., urban, rural).

EMA surveys

For the purpose of this study the EMA surveys were used to calculate compliance with the EMA protocol. The number of completed morning and prompted surveys were used to calculate compliance. In addition, the number of completed binge surveys both overall and per person was calculated.

Interview

At the end of the survey, participants provided qualitative feedback through a 1-hour in-depth interview via an online video-conferencing platform. This interview aimed to get feedback from participants on the general study procedures and smartphone questions. Questions regarding the smartphone app examined participants’ experiences with downloading the RealLife Exp app and installing the survey (e.g., “Did you have any problems with finding or installing the app?”). Then, questions regarding the instructional videos that described study procedures were referenced (e.g., “Are there other materials you think we should provide to help you understand the study procedures and process better?”). In addition, as part of this interview, participants were asked to express their impressions on EMA questions related to identity, appearance-related pressures, and binge eating, and whether the EMA questions captured their overall experiences. Participants were also asked about their experience with completing the binge eating surveys (e.g., “Was it hard to remember to do the binge surveys? Is there anything we could do to help make this easier?”). A copy of the interview guide, including the smartphone questions participants were asked embedded within the guide, can be found in the supplement online (available at https://cdn.amegroups.cn/static/public/mhealth-23-16-1.pdf).

End of study survey

An end of study survey included both quantitative and qualitative questions to gather additional feedback on the study procedures and participant experiences. The following items used a 7-point scale to indicate the feasibility of the binge eating EMA surveys [e.g., “Did you think the length of the morning survey was:” (0= too short to 6= too long)]. Participants also rated their experiences identifying binge eating episodes and completing the binge eating survey after every episode on a 5-point scale (1= disagree to 5= agree). Then, participants reported whether the surveys captured their overall experiences [e.g., “Do you think that there were other identity-related experiences that we did not ask about, that you would have liked us to ask about?” (yes or no)]. In addition, participants reported their physical and mental health state using a 7-point scale [e.g., “I would like to improve my physical health” (1= strongly disagree to 7= strongly agree)]. Lastly, questions asked about participant’s willingness to use mobile technology to track physical and mental health through a 7-point scale [e.g., “I am willing to use mobile technology for tracking my health” (1= strongly disagree to 7= strongly agree)].

Statistical analysis

All interviews were transcribed, and each was independently reviewed by two coders using NVivo 12 software (https://qsrinternational.com/nvivo/nvivo-products/). An initial codebook was developed based on the interview guide (see the supplement online for the interview guide; available at https://cdn.amegroups.cn/static/public/mhealth-23-16-1.pdf), with a focus on: (I) feedback on the training videos, smartphone app, and survey length; (II) feedback on survey content related to binge eating, identity, and appearance-related pressure; (III) suggestions for changes in the general study procedures, training videos, and EMA question wording and meaning. The coders met on a weekly basis to ensure consistency across their analysis of the recorded interviews, and any coding discrepancies were discussed and resolved through consensus. Quantitative data from the demographic questionnaire and end of study survey were analyzed with descriptive statistics calculated in SPSS 26.0 (IBM Corp., Armonk, NY, USA). Compliance rates for the morning and prompted surveys were calculated and descriptive statistics for the number of EMA binge eating surveys completed were calculated. Qualitative and quantitative feedback and suggestions were reviewed together by the team to identify revisions to the study process and survey content, including identifying any unique suggestions that differed between the sexual minority and heterosexual participants.

Results

EMA compliance rates and binge survey frequency

Participants completed an average of 83.7% of the morning surveys and 80.0% of the prompted surveys. The mean number of binge surveys completed was 1.81 (median =2, range =0 to 7), with 17 (81.0%) participants reporting at least one binge episode during the 1-week EMA period.

Study feedback results

Feedback on the training videos, smartphone app, and survey length was compiled from the in-depth interviews and end of study survey and is presented in Table 2. Participants generally found the instructional videos describing the study to be clear and helpful and they similarly thought that downloading the smartphone app from an app store was easy to do. Regarding the survey length, participants on average indicated the morning survey was an appropriate length. Participants that commented on the end of study survey and interview stated that the prompted survey was a bit too long and described the binge survey as being too short (i.e., there was room to accommodate more questions). There were no notable differences in patterns of responding between the heterosexual and sexual minority participants.

Participant feedback on the survey content related to binge eating, identity, and appearance-related pressure from both the interview and survey are presented in Table 3. Regarding the binge eating survey and content, participants expressed an understanding of the definition of binge eating that was provided in the survey. Mean ratings of their ability to identify when binge eating were between 3.86 (heterosexual) and 4.00 (sexual minority) and completion of the binge survey after binge eating agreement was rated between 3.29 (heterosexual) and 4.00 (sexual minority) indicating that they generally agreed they were able to both identify binge episodes and remember to complete the binge eating survey (where 4 is agree; 1= strongly disagree, 5= strongly agree; see Table 3). However, as is illustrated in both the illustrative quotes and the mean ratings not endorsing strong agreement (anchor rating of 5), participants indicated some trouble either identifying that they were binge eating or remembering to complete the survey, an issue we return to when considering recommendations for changes. In collecting feedback on the specific questions included in the binge eating survey, participants reported the questions captured their experiences, but many also noted that this survey was short and identified other questions that could be asked in these surveys (specific suggestions for changes are described below). For the identity-related questions, participants reported they were able to understand the definition described and how to apply it to answer the survey questions, that the identities included in the list were inclusive, and that the questions captured their daily experiences (see Table 3). For the appearance-related pressure content, participants understood the definition and questions, but did note some questions that were confusing and highlighted additional sources of appearance-related pressures that were not fully captured in the survey, as illustrated in the interview quotes. Across all the feedback on survey questions, there were no clear patterns of responding that differed between the sexual minority and heterosexual participants.

Recommendations for changes

Suggestions for changes to the study procedures, training materials, and EMA survey questions were compiled from the interviews and end of study survey responses. These were all reviewed by the research team to make decisions about changes to study procedures or materials. Table 4 provides a summary of the key recommendations made by participants, the source of those recommendations (interview, end of study survey), and a description of the changes that were made for the full study in response to the participant feedback. As with feedback described above, there were no clear patterns of recommended changes based on sexual identity.

Participants’ suggestions for changes that were incorporated into the revised study generally fell into five categories, represented in Table 4, including recommendations for: general study procedure, training videos, binge eating question, identity-related stressor questions, and appearance-related pressure questions. For the general study procedures, we created a frequently asked questions page in the app, extended time participants had to respond to prompts from 30 to 60 minutes, added additional email check-ins from our team, added an end of day survey, and reduced the length of the prompted survey. For the training videos we added a transcript for the videos, added more specific study information, and created a separate informational video about the binge eating survey. This latter addition was made based on quantitative and qualitative feedback and speaks to the difficulty in defining and clearly communicating with participants about what qualifies as a binge eating episode. Within the binge eating survey, participants expressed interest in providing us with more information about the context for, and reactions to, their binge eating; thus, we added 11 additional questions to the binge eating survey (see the supplement for questions added; available at https://cdn.amegroups.cn/static/public/mhealth-23-16-1.pdf). We also made some minor revisions to the identity-related stressor questions, including adding one additional experience and identity (disability) to the list of questions (see supplement; available at https://cdn.amegroups.cn/static/public/mhealth-23-16-1.pdf). Finally, we made substantial revisions to the appearance related pressure questions based on participant feedback. Although these EMA items were based on the trait measure of the Sociocultural Attitudes Towards Appearance Questionnaire-4 Revised (SATA-Q4R), during this study we realized that our adaptation was inadequate for EMA administration. The revised questions are provided in the supplement (available at https://cdn.amegroups.cn/static/public/mhealth-23-16-1.pdf), and include first asking about perceived pressure from four types of people (family members, peers, people of authority, strangers), and then asks about the mode of pressure (in person, or media/social media). Details regarding the revisions made to the study procedures and questions are in Table 4 and in the supplement (available at https://cdn.amegroups.cn/static/public/mhealth-23-16-1.pdf).

Discussion

The present study was a formative, mixed methods pilot study that was conducted to elicit feedback on proposed methods to be used in a large EMA study of binge eating among sexual minority and heterosexual young women. The goal of the larger study is to examine the real-world affective (e.g., negative mood, stress), social (e.g., social interactions, comparisons), health behavior (e.g., alcohol, physical activity), and minority-specific (e.g., sexual minority stressors) factors that may contribute to binge eating among young adult young women who identify as either heterosexual or sexual minority (i.e., cisgender women attracted to women). Given the dearth in EMA research examining binge eating in real time, especially among sexual minority women, this study was critical for providing insight into the optimal training materials, informing general study procedures, and refining EMA question wording. The quantitative and qualitative feedback supported the general feasibility of this EMA protocol; participants were also instrumental in identifying key areas in our study protocols and materials warranting revisions.

One area where we particularly sought feedback was on the EMA surveys that inquired about binge eating, specifically when participants were asked to complete a self-initiated survey each time they binge ate. As expected, some participants reported having difficulty identifying binge episodes for themselves and remembering to complete the survey, which emerged both in quantitative ratings at the end of the study and in the in-depth interviews. This finding that binge episodes can be challenging to identify and record in the moment is consistent with previous studies (29,30). Based on this feedback, for the larger EMA study we adjusted the language describing and defining binge eating that was embedded within the survey and created a separate training video about binge eating for participants to view during the consent process, and it is also available within the EMA survey app during the study. To assist with remembering to self-initiate the binge eating survey, we also include reminders about it in each of the five daily prompted surveys, and in the end of day survey. One unexpected finding was that participants thought that the binge eating survey was too short, and they identified a wide range of topics related to their binge eating that they wanted to share with us in that survey. Based on this feedback, we expanded the binge survey from a set of three questions (time of episode, location, whether with others) to include questions about their perceived trigger(s) and features of the episode (speed, fullness, emotions after, purging/intended purging, mood before, during, and after episode). Although we were trying to limit the questions to reduce burden and improve compliance, participants overwhelmingly reported they were interested in reporting these details as part of the study, as they saw it as potentially helpful for improving our understanding of the factors that may contribute to binge eating.

One other survey content area where we sought feedback during the in-depth interviews was regarding our assessment of sociocultural appearance pressures. When initially designing this set of EMA questions, we modeled the questions after the pressure subscales on the Sociocultural Attitudes Towards Appearance Questionnaire (SATAQ-4R) (31), which assesses pressures from family, peers, significant others, and the media. We adapted questions to be brief and appropriate for administration multiple times per day. Based on the interview feedback, we came to realize that this adaptation for EMA was insufficient. In particular, participants who did not currently have a significant other could not answer that question and noted other people from whom they experienced pressure in their daily lives (e.g., authority figures, strangers, co-workers), which were not captured with our original options. They also questioned whether media was inclusive of social media and identified complex situations that were hard to capture appropriate with our initial question set (e.g., if perceived pressure from a family member on social media, should this be family or media?). Based on this feedback, we reconceptualized our EMA measure of sociocultural pressures to first assess types of people from whom pressure was felt (family members, peers, people of authority, strangers), followed by the format of communication (in person vs. media/social media). This example highlights the importance of pilot testing novel EMA items, consistent with previous calls for quantitative and qualitative testing on EMA item content (8,9). In particular, as shown in Table 4, many of the changes we chose to make for the larger study came from qualitative feedback, underscoring the importance and utility of this mixed methods approach, as suggested by Arigo and colleagues (8). Although the quantitative feedback was helpful for providing ratings of acceptability and feasibility, the qualitative feedback in particular was helpful for identifying whether something was viewed as not ideal, or identifying how to make it better.

The quantitative and qualitative findings from this study both suggested that the prompted survey was too long. In the early interviews, participants also noted that they would often binge eat later in the evening, which is common with binge eating (32), and they would then forget to complete a binge eating survey because the prompted surveys were done for the day. Based on this feedback early in the interviews, in later interviews we probed whether adding a brief survey before participants went to bed would be useful for reminding them about the binge eating survey; this suggestion was largely supported by participants. In the revised protocol for the larger study, we include a before bed EMA survey, where we were able to move some questions that were previously included in the prompted survey and include a reminder about the binge eating survey. Thus, this additional daily questionnaire served several purposes and allowed use to address multiple concerns raised by participants at once.

Sexual minority and heterosexual young women participated in both this formative work and are participating in the larger study. Thus, we were considering any differential patterns in responding based on women’s sexual identity. We did not see any clear evidence that particular materials, procedures, or questions were more or less acceptable to either group. One reason this may be the case is that when designing the EMA questions, we did so in such a way that all the questions were applicable to all participants. For example, when initially selecting measures to assess minority-related stressors via EMA, we considered using measures specifically developed for sexual minority individuals (33-35) for the sexual minority women in our study, but then would also need to identify suitable alternative items for the heterosexual women. Therefore, we instead opted to craft questions based on a study by Panza and colleagues (27) and similar to the Everyday Discrimination Scale (36) format, where we first inquired about a range of experiences, and then asked about aspects of one’s identity that contributed to these experiences. Based on this preliminary work, although participants had some minor suggestions for revisions that we incorporated (see Table 4), it appeared that these questions were generally well received by participants based both on the quantitative and interview feedback (see Table 3).

In Table 4, we present the key changes that we made to the study moving forward, based on the present study. However, it should also be noted that there were recommendations for changes that participants made that we did not use in the revision process. The reasons for not making these changes generally fell into three categories. First, some suggestions, although potentially interesting components for future EMA work, were outside the scope of the scientific questions we aimed to answer in the study. For example, during the interviews some participants suggested adding food and exercise logs throughout the day or adding a survey for people to complete every time they had an urge to binge eat. Second, because we were using an “off the shelf” EMA program, we were technologically limited to the software we selected and unable to accomplish some suggested revisions. For example, several participants suggested having customizable schedules for participants (instead of the 9 am to 9 pm schedule) and adding a progress bar on the surveys. Although these would both have been nice features to add, these were not feasible with the app software at the time. Third, some of the suggestions for revisions appeared ideographic and it seemed unlikely they would have widespread appeal. For example, one participant suggested we create a video of how to download an app from the app store and use the app; based on quantitative ratings of the ease of app use, and the vast majority of participant interview feedback, we did not opt to make this addition. There were also instances where we received conflicting feedback (e.g., some participants wanted the morning survey reminder earlier than 10 am, others wanted it much later), and in such cases we tried to make the decision that would be best suited for most people.

Strengths and limitations

Although this study had many strengths, including its use of a mixed methods approach to evaluate feasibility and acceptability of using EMA methods for understanding binge eating among an at-risk group of young women, there are also limitations that need to be acknowledged. First, this was a small pilot feasibility study and thus we recruited small samples of sexual minority and heterosexual young women. It was important to have individuals with diverse sexual identities to gain varying perspectives, but this study was not powered to quantitatively test for differences in feasibility or acceptability between these two groups (because this was not the primary study aim). Second, most participants in our sample were White and non-Latina, which made it difficult to ensure that the feedback received was from sufficiently diverse racial/ethnic voices. In the larger study, our goal is to more heavily over-recruit racial minority participants, particularly Black women, which will be an important contribution to the disordered eating literature. Third, because of the nature of the data collection process (i.e., initially not having any eligible sexual minority women sign up through local recruitment, then needing to pivot to fully virtual national data collection due to the COVID-19 pandemic), the recruitment sources varied across sexual minority versus heterosexual women. However, because there were no observed differences across these groups, attribution of differences was not an issue. Moreover, the larger study currently underway is using virtual recruitment for all participants. Finally, participants had to be willing to use a smartphone to complete an EMA protocol, which can be intensive, and some people may not be willing to do.

Conclusions

This mixed methods study was designed to collect formative data regarding the practicality and acceptability of a larger EMA study focused on the binge eating behaviors of young sexual minority and heterosexual women. Although many aspects of the design were rated as acceptable and appropriate, based on participant recommendations, changes were made to the procedures and training materials to increase compliance (e.g., email encouragement) and understanding of participant tasks (e.g., reasoning behind random prompts). As binge eating is the central focus of the larger study, revisions were also made to provide clarity on the definition of binge eating, increase completion of the self-initiated survey following binge episodes, and provide more context around binge eating episodes. Overall, this study emphasizes the importance of soliciting both quantitative and qualitative feedback from participants in the target sample prior to the larger study and provides evidence confirming the feasibility of a larger EMA study.

Acknowledgments

A portion of this manuscript was presented at the annual meeting of the Society of Behavioral Medicine in April 2021 and the abstract from this presentation was published in the 2021 Annals of Behavioral Medicine Annual Meeting Abstracts Supplement.

Funding: Research reported in this publication was supported by the National Institute of Minority Health and Health Disparities of the National Institutes of Health (No. R01MD012598 to KE Heron, AL Braitman, and RJ Lewis). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Footnote

Reporting Checklist: The authors have completed the STROBE reporting checklist. Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-16/rc

Data Sharing Statement: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-16/dss

Peer Review File: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-16/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-16/coif). KEH serves as an unpaid editorial board member of mHealth from March 2023 to February 2025. She also received funding from National Institute on Minority Health and Health Disparities (NIMHD) of the National Institutes of Health (NIH) to conduct this study and for related activities (No. R01MD012598). KEH is also a member of the Executive Committee for the Society of Ambulatory Assessment. ALB and RJL received funding as co-investigators from NIMHD of NIH (No. R01MD012598). The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Intuitional Review Board of Old Dominion University (No. 1493285) and informed consent was obtained from all participants.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Mehl MR, Conner TS. editors. Handbook of Research Methods for Studying Daily Life. The Guilford Press; 2012.
2. Silvia PJ, Cotter KN. Researching daily life: A guide to experience sampling and daily diary methods. American Psychological Association; 2021.
3. Soyster PD, Fisher AJ. Involving stakeholders in the design of ecological momentary assessment research: An example from smoking cessation. PLoS One 2019;14:e0217150. [Crossref] [PubMed]
4. Burke LE, Shiffman S, Music E, et al. Ecological Momentary Assessment in Behavioral Research: Addressing Technological and Human Participant Challenges. J Med Internet Res 2017;19:e77. [Crossref] [PubMed]
5. Forkmann T, Spangenberg L, Rath D, et al. Assessing suicidality in real time: A psychometric evaluation of self-report items for the assessment of suicidal ideation and its proximal risk factors using ecological momentary assessments. J Abnorm Psychol 2018;127:758-69. [Crossref] [PubMed]
6. Braitman AL, Romano KA, Heron KE, et al. Psychometric analysis and validity of the Daily Sexual Minority Stressors Scale among young adult same-sex female couples. Psychol Assess 2021;33:1025-37. [Crossref] [PubMed]
7. Heron KE, Braitman AL, Lewis RJ, et al. Measuring Sexual Minority Stressors in Lesbian Women's Daily Lives: Initial Scale Development. Psychol Sex Orientat Gend Divers 2018;5:387-95. [Crossref] [PubMed]
8. Arigo D, Mogle JA, Brown MM, et al. A multi-study approach to refining ecological momentary assessment measures for use among midlife women with elevated risk for cardiovascular disease. Mhealth 2021;7:53. [Crossref] [PubMed]
9. Boesen VB, Nissen SB, Groenvold M, et al. Conversion of standard retrospective patient-reported outcomes to momentary versions: cognitive interviewing reveals varying degrees of momentary compatibility. Qual Life Res 2018;27:1065-76. [Crossref] [PubMed]
10. Doherty K, Balaskas A, Doherty G. The design of ecological momentary assessment technologies. Interact Comput 2020;32:257-78. [Crossref]
11. Christensen TC, Barrett LF, Bliss-Moreau E, Lebo K, Kaschub C. A practical guide to experience-sampling procedures. J Happiness Stud 2003;4:53-78. [Crossref]
12. Degroote L, DeSmet A, De Bourdeaudhuij I, et al. Content validity and methodological considerations in ecological momentary assessment studies on physical activity and sedentary behaviour: a systematic review. Int J Behav Nutr Phys Act 2020;17:35. [Crossref] [PubMed]
13. Heron KE, Braitman AL, Dawson CA, et al. Evaluating study procedure training methods for a remote daily diary study of sexual minority women. Mhealth 2021;7:46. [Crossref] [PubMed]
14. Carpenter C. Sexual orientation and body weight: Evidence from multiple surveys. Gend Issues 2003;21:60-74. [Crossref]
15. Conron KJ, Mimiaga MJ, Landers SJ. A population-based study of sexual orientation identity and gender differences in adult health. Am J Public Health 2010;100:1953-60. [Crossref] [PubMed]
16. Struble CB, Lindley LL, Montgomery K, et al. Overweight and obesity in lesbian and bisexual college women. J Am Coll Health 2010;59:51-6. [Crossref] [PubMed]
17. Austin SB, Ziyadeh NJ, Corliss HL, et al. Sexual orientation disparities in purging and binge eating from early to late adolescence. J Adolesc Health 2009;45:238-45. [Crossref] [PubMed]
18. Heffernan K. Eating disorders and weight concern among lesbians. Int J Eat Disord 1996;19:127-38. [Crossref] [PubMed]
19. Ivezaj V, White MA, Grilo CM. Examining binge-eating disorder and food addiction in adults with overweight and obesity. Obesity (Silver Spring) 2016;24:2064-9. [Crossref] [PubMed]
20. Smyth JM, Wonderlich SA, Heron KE, et al. Daily and momentary mood and stress are associated with binge eating and vomiting in bulimia nervosa patients in the natural environment. J Consult Clin Psychol 2007;75:629-38. [Crossref] [PubMed]
21. Thompson MA, Heinberg LJ, Altabe M, et al. Exacting Beauty: Theory, Assessment, and Treatment of Body Image Disturbance. American Psychological Association; 1999.
22. Harrison K, Cantor J. The relationship between media exposure and eating disorders. J Commun 1997;47:40-67. [Crossref]
23. Stice E. A prospective test of the dual-pathway model of bulimic pathology: mediating effects of dieting and negative affect. J Abnorm Psychol 2001;110:124-35. [Crossref] [PubMed]
24. Crandall KJ, Eisenman PA, Ransdell L, et al. Exploring binge eating and physical activity among community-dwelling women. Int J Psychol Behav Sci 2011; [Crossref]
25. Gadalla T, Piran N. Co-occurrence of eating disorders and alcohol use disorders in women: a meta analysis. Arch Womens Ment Health 2007;10:133-40. [Crossref] [PubMed]
26. Mason TB, Lewis RJ, Heron KE. Daily discrimination and binge eating among lesbians: A pilot study. Psychol Sex 2017;8:96-103. [Crossref]
27. Panza E, Fehling KB, Pantalone DW, et al. Multiply marginalized: Linking minority stress due to sexual orientation, gender, and weight to dysregulated eating among sexual minority women of higher body weight. Psychol Sex Orientat Gend Divers 2021;8:420-8. [Crossref] [PubMed]
28. Smyth J, Wonderlich S, Crosby R, et al. The use of ecological momentary assessment approaches in eating disorder research. Int J Eat Disord 2001;30:83-95. [Crossref] [PubMed]
29. Stein KF, Corte CM. Ecologic momentary assessment of eating-disordered behaviors. Int J Eat Disord 2003;34:349-60. [Crossref] [PubMed]
30. Wilfley DE, Schwartz MB, Spurrell EB, et al. Assessing the specific psychopathology of binge eating disorder patients: interview or self-report? Behav Res Ther 1997;35:1151-9. [Crossref] [PubMed]
31. Schaefer LM, Harriger JA, Heinberg LJ, et al. Development and validation of the sociocultural attitudes towards appearance questionnaire-4-revised (SATAQ-4R). Int J Eat Disord 2017;50:104-17. [Crossref] [PubMed]
32. Smyth JM, Wonderlich SA, Sliwinski MJ, et al. Ecological momentary assessment of affect, stress, and binge-purge behaviors: day of week and time of day effects in the natural environment. Int J Eat Disord 2009;42:429-36. [Crossref] [PubMed]
33. Heron KE, Lewis RJ, Shappie AT, et al. Rationale and Design of a Remote Web-Based Daily Diary Study Examining Sexual Minority Stress, Relationship Factors, and Alcohol Use in Same-Sex Female Couples Across the United States: Study Protocol of Project Relate. JMIR Res Protoc 2019;8:e11718. [Crossref] [PubMed]
34. Mohr JJ, Sarno EL. The ups and downs of being lesbian, gay, and bisexual: A daily experience perspective on minority stress and support processes. J Couns Psychol 2016;63:106-18. [Crossref] [PubMed]
35. Totenhagen CJ, Randall AK, Cooper AN, et al. Stress spillover and crossover in same-sex couples: Concurrent and lagged daily effects. J GLBT Fam Stud 2017;13:236-56. [Crossref]
36. Williams DR, Yu Yan, Jackson JS, et al. Racial Differences in Physical and Mental Health: Socio-economic Status, Stress and Discrimination. J Health Psychol 1997;2:335-51. [Crossref] [PubMed]