More than juxtaposition: a commentary to Willis et al.’s (2023) mixed method study on mobile mental health interventions

In our search for existing literature on mental health interventions for Black, Indigenous, and People of Color, we read with great interest the article by Willis and Neblett (1) published in January 2023. This article aimed to obtain data for developing mobile health interventions for Black people with one or more mental health disorders. We believe that the themes discussed in Willis and Neblett (1) are highly relevant to the current mental health care context in which Black, Indigenous, and People of Color experience a complex intersection of stressors related to racialization and stigmatization of mental disorders (2). Using a mixed-methods design, the authors highlight that sociocultural experiences of racialization play a key role in Black people’s attitudes toward mental health and the use of mobile mental health interventions. The authors also highlight Black people’s experiences with existing mobile interventions, including their strengths and limitations, and identify desired characteristics for a future culturally competent mobile intervention. Yet, despite the relevance of the themes discussed by Willis and Neblett (1), we have uncovered various methodological limitations that raise questions, particularly regarding the lack of information on certain methodological aspects related to mixed-method studies, as well as the instruments used to measure participants’ attitudes toward mobile health interventions.

First, Willis and Neblett (1) report conducting a mixed-method study. According to Pluye and Hong (3), a mixed-methods study is defined as a research design that joins the strengths of both quantitative and qualitative methodologies to compensate for their respective limitations. While the quantitative and qualitative phases are well articulated, including their respective purposes, designs, and data collection and analysis methods, Willis and Neblett (1) do not specify the reasons for, or value of, using a mixed design in their study. In the methodology section, the authors justify the choice to conduct a mixed-methods study by indicating the advantages of qualitative data collection methods, i.e., focus groups, as well as the advantages of using questionnaires to collect quantitative data. While we would like to acknowledge the relevance of justifying the data collection methods used (4), this section of the article does not seem to provide sufficient justification for conducting a mixed-methods study. To this end, the Good Reporting of A Mixed Methods Study (GRAMMS) (5) guidelines, and more recently, Fàbregues et al. (6), in their methodological review of mixed studies, suggest explicitly stating the reasons for using a mixed design. This step is essential when publishing mixed studies since it allows the relevance and contribution of the combination of approaches to be validated in comparison with the use of the two designs in parallel (7). Similarly, although it seems to be common in mixed studies 6), Willis and Neblett (1) do not explicitly present the design used in their mixed study. However, according to GRAMMS (5) and more recently, Younas et al. (8), the identification and explicit description of the mixed design used (i.e., sequential explanatory design, sequential exploratory design, concurrent design) greatly facilitates the evaluation of the methodological quality of the study, particularly regarding the priority of the designs and the role of the non-dominant design, if any.

Furthermore, a central criterion of mixed designs is the integration of quantitative and qualitative phases. Although this is an essential component of mixed methods studies (3,5,8), the authors do not explicitly state the preferred method(s) of integration for combining the data, analysis, or results of the quantitative and qualitative phases. As Pluye and Hong (3) point out, juxtaposing quantitative and qualitative designs in the same study without using an integration method does not constitute a mixed research design. Further, the lack of integration or inadequate integration of quantitative and qualitative phases can compromise the study’s rigor and its conclusions (8). To this end, the GRAMMS guidelines (5) recommend describing when the integration occurred, how it was accomplished, and who was involved. Explaining the method(s) of integration could have guided the interpretation of the results and the resulting recommendations, as well as facilitated critical evaluation of the quality of the writing by the readers.

Second, the authors report that they developed an instrument for this study to measure participants’ attitudes toward using mobile mental health interventions (1). Although we acknowledge the intention of the authors to develop a new instrument, little information is available about the instrument development process and the analyses undertaken to assess its validity and reliability before this study. According to Polit and Yang (9), prior validation of an instrument is essential to ensure the research’s validity, credibility, and inferences. Indeed, the recommendations from this study may not be consistent for Black people living with a mental health disorder. In addition, the availability of data from the evaluation of the instrument may allow for its use or adaptation in other research projects, thus contributing to its dissemination.

In light of these comments, we are compelled to note that it is difficult to appreciate the methodological quality of Willis and Neblett’s study (1) and the results and inferences derived from it. We believe that the reflections generated by this paper will contribute to discussions about the challenges of reporting mixed research design methodology and the use of measurement scales in quantitative research. We believe it is imperative to strengthen the methodological quality of mixed-methodology health studies as the combined strengths of quantitative and qualitative studies can provide a deep and nuanced understanding of the experience of Black, Indigenous, and People of Color in mental health care.

Acknowledgments

We would like to aknowledge the advices and recommendations of Professor Marilyn Aita in writing this manuscript.

Funding: None.

Footnote

Provenance and Peer Review: This article was a standard submission to the journal. The article did not undergo external peer review.

Conflicts of Interest: Both authors have completed the ICMJE uniform disclosure form (available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-23-26/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Willis HA, Neblett EW. Developing culturally-adapted mobile mental health interventions: a mixed methods approach. Mhealth 2023;9:1. [Crossref] [PubMed]
2. Taylor D, Richards D. Triple Jeopardy: Complexities of Racism, Sexism, and Ageism on the Experiences of Mental Health Stigma Among Young Canadian Black Women of Caribbean Descent. Front Sociol 2019;4:43. [Crossref] [PubMed]
3. Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health 2014;35:29-45. [Crossref] [PubMed]
4. Polit DF, Beck CT. Nursing Research: Generating and assessing evidence for nursing practice. 11th ed. Philadelphia: Wolters Kluwer; 2021.
5. O'Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy 2008;13:92-8. [Crossref] [PubMed]
6. Fàbregues S, Mumbardó-Adam C, Escalante-Barrios EL, et al. Mixed methods intervention studies in children and adolescents with emotional and behavioral disorders: A methodological review. Res Dev Disabil 2022;126:104239. [Crossref] [PubMed]
7. Fetters MD. Chapiter 1: Identifying a Topic, Rationale, and Potential Feasibility Issues for Conducting a Mixed Methods Research of Evaluation Project. The Mixed Methods Research Workbook: Activities for designing, implementing, and publishing projects. Thousand Oaks, California: SAGE Publications 2020.
8. Younas A, Pedersen M, Tayaben JL. Review of Mixed-Methods Research in Nursing. Nurs Res 2019;68:464-72. [Crossref] [PubMed]
9. Polit DF, Yang F. Chapter 4 – Challenges in scale development. Measurement and the measurement of change: A primer for the health professions. Wolters Kluwer; 2016.