Implementation of a computer-assisted cognitive-behavioral therapy program for adults with depression and anxiety in an outpatient specialty mental health clinic

Introduction

Psychotherapy is an effective form of mental health treatment for depression (1) and is frequently preferred by patients (2). However, access to high-quality psychotherapy remains out of reach for many people who are struggling (3). Frequent barriers to psychotherapy include cost and excessive wait times (4,5). Technology solutions to augment or extend mental health care have potential to address these barriers and increase access to evidence-based psychotherapies (6-10). One approach is the use of computer-based modules to teach psychoeducational and skills-based psychotherapy content (11). Although computer-based programs that deliver psychotherapy content without a live psychotherapy or support component have shown low levels of patient engagement (4,5), engagement is substantially increased when live clinical or coaching support is provided along with the online content (6,7,9,10). This combination of computer-based content and live support is often referred to as computer-assisted psychotherapy. Cognitive-behavioral therapy (CBT) is a widely-used and effective psychotherapy for a range of mental health issues (12) and is well-suited for adaptation to a computer-assisted format due to the CBT focus on skills and psychoeducation. Meta-analyses examining nearly 15 years of research show that computer-assisted CBT (cCBT) programs can be as efficacious as traditional CBT (7,11,13) and that treatment effects may be largest for non-primary care settings (10).

In terms of acceptability, patient and clinician attitudes toward technology-based mental health interventions are somewhat mixed. However, both groups demonstrate at least a subset of people with interest and willingness to utilize the interventions. For example, some patients hold favorable attitudes toward mental health interventions that incorporate technology (14) and they are willing to try technology based interventions (15), especially when a desired treatment is scarce or otherwise unavailable to them (16). Among clinicians, one survey showed concerns about the potential negative impact on patient rapport and uncertainty whether the programs would be efficacious in reducing mental health symptoms but also favorable views about utilizing cCBT and a sense they had the technical skills need to integrate the programs (17).

Because cCBT programs are shown to be efficacious, have demonstrated at least moderate patient and clinician acceptability, and have strong potential to increase access to care, we implemented a cCBT program for adults as a new clinical offering in our specialty mental health clinic in a large academic medical center. The program was implemented to improve access to psychotherapy services in our setting. We implemented the program as a clinical quality improvement effort, with the goals of offering timely access to psychotherapy services and demonstrating good uptake, engagement and effectiveness at reducing symptoms. We are aware of no other published descriptions of developing and implementing this type of program in a real-world setting. Therefore, we also describe in detail our experiences, outcomes and lessons learned during the design and implementation of the program.

Methods

Context

The cCBT program was implemented in an outpatient specialty mental health clinic focusing on mood and anxiety disorders that is located on a large, urban academic medical campus. The clinic conducts more than 15,000 visits and treats more than 1,000 unique patients annually. Approximately 75% of clinic visits are conducted by telehealth. The primary services offered are psychiatric medication management and individual psychotherapy. Of note, psychotherapy referrals from psychiatric prescribers in the clinic are prioritized ahead of new-to-clinic callers and these internal psychotherapy referrals tend to fill or exceed available psychotherapy capacity. Thus, traditional psychotherapy services are often not available to new callers, which was the case during the period of time the cCBT program was offered.

Among clinic patients, a large majority of patients in the clinic identify as White/Caucasian and non-Hispanic or Latino/a. Nearly all patients identify English as their primary language. Many patients are privately insured with a small proportion having government payors. The clinic serves all ages, although the cCBT program was offered only to patients aged 18 years and older. The cCBT program was designed and launched in the clinic after receiving institutional funding to support pilot programs to increase access to mental health services during the COVID-19 pandemic.

Developing an industry partnership

Selecting and partnering with a vendor to provide the computer portion of the program was the first step in developing our cCBT program. We selected the Good Days Ahead program offered through MindStreet, Inc. due to its established efficacy for our primary patient population (patients with depression, anxiety, and/or mood disorders), its HIPAA-compliant platform and the inclusion of patient and clinician dashboards. Upon our initial decision to pursue the use of Good Days Ahead, we contacted MindStreet, Inc. and negotiated a reduced licensure fee for our pilot program. We then worked to obtain additional approvals required by our institution (e.g., technology and data-sharing approvals, business associate agreement). This approval process included multiple rounds of negotiation and took approximately 9 months.

Intervention description

Access to the cCBT program was available through two distinct pathways. The first pathway was designed for patients new to the clinic, who were not otherwise able to access psychotherapy services during the time period of the cCBT program. In this pathway, potential patients contacting the clinic to establish care could opt to engage in a time-limited course of cCBT combined with regular 30-min telehealth appointments with a clinician. We refer to this pathway as “cCBT-only”. The second pathway was for patients already established in psychotherapy in the clinic and is referred to as the “augmented-psychotherapy” pathway. In augmented-psychotherapy, clinicians identified patients already established with them who they believed may benefit from the online CBT modules and invited them to access the online modules. The augmented-psychotherapy pathway was not time-limited and somewhat less structured than the cCBT-only pathway in order to facilitate integration with existing ongoing care processes. However, because the cCBT-only program involved the introduction of a new care pathway from start to finish in the clinic, several overarching goals informed design of the cCBT-only pathway.

• Provide timely patient access to treatment. A primary aim was to provide timely access for new patients to begin treatment. We designed care in the cCBT-only pathway as time-limited to generate consistent patient flow and clinician availability for new patients. To enable and support the time-limited structure, patients were notified up front of the time-limited nature of the program and external referral resources were provided when patients completed the program and required or desired additional treatment. Clinicians protected time each week to evaluate new patients and get them started in the program and for ongoing 30-min telehealth appointments with cCBT-only patients.
• Adhere to evidence-based practice. We selected a vendor with an evidence-based product and included an initial brief screen with a patient staff coordinator and a second 45-min screening appointment with a clinician to ensure that patient presenting issues were within the scope of the product. In addition, due to the robust finding that computer-based psychotherapy programs are most beneficial when they include a live component (6), we offered 30-min individual telehealth appointments with a clinician. Patients were strongly encouraged to use these appointments and could schedule as often as once weekly but typically scheduled appointments every 2–3 weeks.
• Create a program that is well accepted by patients and clinicians. To identify patients that we expected to benefit from and be satisfied with the program, we invested time in training so that clinic staff could accurately describe the program to potential patients and best support them in deciding whether to pursue the program. We also had clinicians protect time to describe the program and answer patient questions at initial appointments before patients started in the program. To identify clinicians who were likely to be satisfied with their work in the program, clinicians opted-in to participate in the program. An overview of the program was presented during clinical team meetings and program clinicians were selected from the subset of clinicians with strong interest in participating.

Computer-assisted content

The computer-assisted portion of the program was provided by the 9-module version of the Good Days Ahead program, which is accessed through a web browser on a computer, tablet or smartphone. The modules are designed to be completed by patients in a sequential format at a recommended pace of 1 module per week. Each module includes written content, clinical vignettes and information presented through videos and interactive quizzes and exercises. Primary areas of content include identifying thoughts and feelings about specific situations, cognitive restructuring and activity scheduling. There is also a companion clinician dashboard that allows clinicians to view patient logins, depression and anxiety self-ratings on a 0–10 scale, patient progress through the modules and patient responses to interactive exercises.

Patient population

The program was designed for adults endorsing depression and anxiety symptoms and seeking care in a depression and anxiety disorders specialty outpatient clinic. The focus on depression and anxiety was communicated to patients or potential patients during the initial program description, which occurred either during a consultation with a current provider or when contacting the clinic for services. However, there were no formal symptom criteria requirements [i.e., “cutoffs” on Patient Health Questionnaire-9 (PHQ-9) or Generalized Anxiety Disorder-7 (GAD-7) scores] to qualify for the program per the intent to use the program in a live clinic setting. The only requirements for program participation were patient age being 18 years or older, ability to speak and read in English (due to the online content being in English and clinicians speaking English) and a high-speed internet connection to access online content. Additionally, for the cCBT-only pathway, our clinical team decided in advance that patients who reported current active suicidal ideation, had experienced a suicide attempt in the past 3 months and/or had an active, primary substance use disorder would be recommended to a higher level of care at the time of the initial program appointment with a clinician. However, no patients who completed the initial appointment were recommended to a higher level of care due to those or other reasons.

Participating clinicians

Participating clinicians were recruited by providing a brief description of the cCBT program and the Good Days Ahead platform during a weekly clinical team meeting. Those interested in participating as a clinician in the program were asked to indicate their interest during the meeting or in a follow up email. Due to a variety of logistical factors (e.g., limited program funding, clinician amount of time spent in clinic versus other duties, availability to attend training) only a subset of clinicians who expressed initial interest were selected as project clinicians.

cCBT-only pathway

Treatment in the cCBT-only pathway was offered at no cost to patients. This included both access to the online module content and the 30-min telehealth appointments. Our ability to offer the program at no cost to patients was enabled by our internal program funding and a discounted price from our partnering vendor. The primary entry for patients into the cCBT-only program was by expressing interest in the program when initially contacting the clinic for services and after receiving a basic program description. The cCBT-only program was briefly described in a recorded message patients heard upon calling the new patient line and was also described by the clinic coordinator when discussing services directly with potential patients. The coordinator typically informed potential patients about wait times and availability for medication management, that standard psychotherapy services were not currently available, and provided a brief description of the cCBT-only program, including that it had minimal wait time and was being offered at no cost to patients. In some cases, callers were unaware of the cCBT program until they contacted the clinic. In other cases, patients called to seek out the program directly after seeing program fliers posted in the clinic or being referred by a clinician who had received a listserv announcement about the program. Interested patients were scheduled for a brief assessment and informational appointment with a cCBT clinician who was either a Licensed Psychologist or Social Work Fellow. At the appointment, the clinician provided a brief overview of the program and answered any patient questions. The clinician assessed level of anxiety and depressive symptoms and screened for factors that might indicate need for a higher level of care. Those patients who remained interested in the cCBT program after discussion with the clinician were sent an email invitation to create a patient account via the Good Days Ahead clinician dashboard. Patients were then matched with a cCBT clinician for the remainder of the program. In most cases, this was the same clinician who had completed the initial appointment although in some cases patients were transferred to another clinician due to patient scheduling preferences. Patients were scheduled for a follow up telehealth check-in appointment with the assigned clinician and encouraged to create an account and complete at least one cCBT module prior to the next scheduled appointment. An overview of the process for entering and starting the cCBT-only pathway appears in Figure 1.

Figure 1 Patient flow into cCBT-only program. cCBT, computer-assisted cognitive-behavioral therapy.

Telehealth appointments focused on review and practice of module content. This included review of patient responses, extension of skills or concepts and guidance in applying skills to specific patient examples. Clinicians were given full discretion to supplement appointments with additional content, handouts or tools beyond those covered in the computer modules if deemed relevant for a patient.

To assess program acceptability, cCBT-only patients were invited to respond to a brief, self-designed, anonymous online survey at program completion or discontinuation. The survey inquired about satisfaction, the degree to which the program met patient needs and hypothetical willingness to pay for the online content and live support components of the program.

Augmented-psychotherapy pathway

In augmented-psychotherapy, clinicians already participating in the cCBT-only program were encouraged to offer Good Days Ahead as an augment to more traditional individual psychotherapy with established patients. In those cases, clinicians identified established patients who they felt may benefit from additional skills building or out-of-session structure and discussed the program with the patients. Interested patients were sent an invitation to register for the online modules and encouraged to complete modules. Under this application of the program, patient appointments with clinicians were typically 50–60 min and consisted of more traditional outpatient psychotherapy. In addition, the live clinician appointments did not end upon completion or discontinuation of the online modules. In this version of the program, patients were provided with online module access free of charge. However, live appointments were subject to the usual fees and/or insurance co-pays associated for clinic psychotherapy appointments. An overview of patient flow through both cCBT-only and augmented-psychotherapy programs appears in Figure 2.

Figure 2 Overview of cCBT-only and augmented-psychotherapy pathways. cCBT, computer-assisted cognitive-behavioral therapy.

Analytic approach

The following metrics were used to evaluate the impact of the program:

• Program uptake defined as the percentage of new clinic callers seeking psychotherapy services who started the cCBT-only program (cCBT-only pathway) and the percentage of patients who started augmented-psychotherapy out of total number of active patients on cCBT clinician caseloads (augmented-psychotherapy pathway) over a 12-month period.
• Patient access (cCBT-only pathway) defined as the mean number of days between a patient expressing interest in the cCBT-only program and the date of the patient’s first scheduled cCBT-only appointment.
• Patient engagement (both pathways) defined as the number and distribution of online modules completed by patients in the program on each pathway.
• Program effectiveness defined as (I) the change from baseline to endpoint in patient anxiety and depression scores on the PHQ-9, GAD-7 and 10-point anxiety and depression self-ratings that patients were prompted to make upon login to the online modules (both pathways); and (II) the percentage of cCBT-only patients who did not desire additional psychotherapy at cCBT-only program completion or discontinuation (cCBT-only pathway).
• Patient acceptability (cCBT-only pathway) defined as the percentage of patients who were satisfied and felt the program met their needs based on patient survey responses.
• Clinician satisfaction was assessed through open-ended questions at regular team meetings. Questions focused on general experiences working in the program and asked specifically about benefits and challenges to working in the cCBT program.

Data regarding patient access, engagement and acceptability are reported as descriptive statistics. Statistical comparison of engagement levels on the cCBT-only vs. augmented-psychotherapy pathways was made using independent samples t-tests in SPSS version 29. Data for program effectiveness were analyzed using paired sample t-tests for the subset of patients in both pathways that had baseline and end-point data for the outcome of interest (e.g., 10-point depression rating, PHQ-9, 10-point anxiety rating, GAD-7). Qualitative data regarding clinician satisfaction are summarized as themes, although formal qualitative data analysis was not conducted on this data.

Ethical considerations

This is an evaluation of the cCBT program that does not meet the definition of research. The evaluation was submitted to the Colorado Multiple Institutional Review Board (COMIRB) and was determined to be quality improvement and not human subject research. It was thus exempt from review.

Results

Program uptake

Twenty-seven patients started the program over a 12-month period with 18 starting in the cCBT-only pathway and 9 starting in the augmented-psychotherapy pathway. In that same time period, the clinic received contacts on the new patient phone line or email from 744 patients seeking new psychotherapy services. A high proportion of these initial contacts did not have any subsequent contact with the clinic as they did not answer a callback from clinic staff and/or did not contact the clinic again after staff left a return voicemail or email. Among those who did answer or recontact clinic staff, many elected to receive outside referrals for psychotherapy. Of the 744, 186 expressed initial interest in the cCBT-only program and 24 completed clinic paperwork for the program. Ultimately, 18 patients started the cCBT-only pathway, which represented just 2.4% of new patients who initially inquired about psychotherapy services. Patient reasons for not pursuing the cCBT-only program were not systematically tracked and systematically tracked. In many cases their reasons for not pursuing the program were unknown because they did not follow up with the clinic after the initial contact in which they heard a preliminary cCBT program description and were informed that traditional psychotherapy services were unavailable. During the same 12-month period, the three clinicians working in the cCBT program used the augmented-psychotherapy pathway with 9 of approximately 85 (10.6%) established clinic patients.

Patient access

On average, patients in the cCBT-only program were scheduled for an appointment within 14 days of confirming interest in the program to clinic staff (mean 13.2 days; range, 2–30 days).

Patient engagement

Descriptives regarding module completion and time in the program appear in Table 1. Patients completed an average of 5.9 modules [standard deviation (SD) =3.4]. An independent samples t-test showed that patients in the cCBT-only pathway completed more modules than those in the augmented-psychotherapy pathway [M (SD)_cCBT-only =6.8 (3.1), M (SD)_{augmented-psychotherapy} =4.0 (3.4), t(25) =2.21, 95% confidence interval (CI): 0.20–5.58, P=0.04]. Module completion across the cCBT-only and augmented-psychotherapy pathways appears in Figure 3. Patients in the cCBT-only pathway accessed an average of 7.2 telehealth appointments during their time in the program (SD =2.9).

Figure 3 Number of patients completing modules in the cCBT program by cCBT-only and augmented-psychotherapy pathways over a 12-month period. Graph represents the number of patients who completed the informational appointment, elected to participate, and completed various numbers of cCBT modules. A larger number of patients participated in the cCBT-only pathway compared to the augmented-psychotherapy pathway. Participation tapered off across the program in both pathways. cCBT, computer-assisted cognitive-behavioral therapy.

Program effectiveness

Table 2 shows descriptive statistics for depressive and anxiety symptoms among all patients, cCBT-only patients and augmented-psychotherapy patients at program start and end or discontinuation, where available. A paired-samples t-test that included the subset of patients with baseline and end point data showed a decrease in depressive symptoms from baseline to endpoint on the PHQ-9, t(17) =4.95, 95% CI: 2.29–5.70, P<0.001, and the 10-point self-rating, t(Jeny24) =3.26, 95% CI: 0.54–2.51, P=0.003. Findings were more variable for anxiety scores. Symptom reductions on the 10-point self-rating were significant, t(Jeny24) =2.38, 95% CI: 0.14–1.94, P=0.003 while ratings on the GAD-7 were not, t(Jeny16) =1.74, 95% CI: −0.52 to 2.53, P=0.26. There was variability in whether patients felt they needed additional psychotherapy or a more traditional form of psychotherapy after program completion or discontinuation (Figure 4).

Figure 4 Patient status at end or discontinuation of cCBT-only program. Pie slices represent the proportion of patients falling into each category at the end or discontinuation of the cCBT-only program. cCBT, computer-assisted cognitive-behavioral therapy.

Acceptability

Twelve of 18 patients (66.7%) on the cCBT-only pathway responded to the patient acceptability survey. Results appear in Figure 5. Acceptability was mixed with most patients (83.3%) indicating they were satisfied with the program but only 41.7% indicating the program fully met their needs. More than half indicated they would be willing to pay to access the online modules and 75.0% indicated they would be willing to pay appliable insurance co-pays or fees to utilize telehealth appointments.

Figure 5 Patient acceptability of cCBT-only program. Graph represents views from patients in the cCBT-only pathway who completed the anonymous survey regarding program participation (n=12; 66.7% survey response rate). Outcomes are reported as proportions of total survey responses. Survey respondents were satisfied with the program but mixed on whether it met their full needs. Slightly more than half indicated they would have been willing to pay to access online modules and 75% would have been willing to pay to access live appointments. cCBT, computer-assisted cognitive-behavioral therapy.

Clinician participation and satisfaction

Eight of 18 (44.4%) clinicians expressed initial interest in participating in the program after hearing about it at a team meeting. Of those interested, three clinicians were selected to participate in the initial program. All three clinicians who were selected to participate identified many positive themes to working in the program. Major positive themes identified by clinicians included finding participation in the program to be more enjoyable than expected and appreciating the structure of the program. For example, clinicians shared that they enjoyed the program because it was a different way to interact with patients, that they saw patient benefits in a relatively short time, and that the structure often required less preparation in advance of an appointment. Providers also appreciated having the focus and structure of reviewing content from modules during live support sessions while also having flexibility to weave in interventions and strategies beyond the cCBT program based on individual patient needs. Clinicians relayed that the small subset of patients who started and actively participated in the program seemed to like and benefit from it. They also noted that patients talked about “learning like a class” and “increasing mental health vocabulary”, which suggested to them that the online modules were helpful for consolidation and retention of concepts and skills and were valuable beyond what evaluation metrics might capture. Lastly, one clinician who was in advanced stage of training shared that the structure of the program helped to further their own learning regarding CBT. The primary negative feedback from clinicians focused on frustration and perceptions of patient frustration related to technical challenges with the online modules (e.g., difficulty logging into user accounts, module progress and/or responses to exercises not saving). Additional general feedback was that clinicians enjoyed participating in the program for a portion of their clinical work but would not want it to comprise all of their clinical work and that patient engagement was substantially higher for cCBT-only than the augmented-psychotherapy pathways. Clinicians also observed that the program seemed most helpful for patients in targeting problems that our selected vendor program was specifically designed to address (e.g., depression). However, the program was less helpful for patient symptoms that were not directly addressed in our selected online content (e.g. insomnia and trauma).

Summary of program outcomes

Our aims were to utilize cCBT to increase access to psychotherapy services in our clinic while demonstrating good uptake, acceptability and effectiveness of the cCBT program. We found that integrating a cCBT program into the day-to-day workflows and structures of an existing clinic provided good psychotherapy access to participants and that depressive symptoms improved during program participation among the very small subset of patients willing to try cCBT. Clinicians were well-satisfied with using cCBT to engage in a portion of their clinical work. However, uptake was low with very few patients opting to participate in the program. Patients who did participate were mixed in their acceptability. We were surprised by the extent to which new patients seeking psychotherapy chose to receive external psychotherapy referrals rather than pursue cCBT. The low uptake of our program suggested that our setting was not well-suited to cCBT and we discontinued the program after 12 months.

Lessons learned and associated recommendations

Based on our experiences with the program, we offer the following lessons learned and associated recommendations:

Thoughtfully consider the setting

We suspect the low uptake in our setting may be due to having a high number of self-referred patients who are looking for more traditional psychotherapy services that include specialized care in an academic medical setting. Carefully considering how cCBT does or does not match patient expectations of the type of treatment they will receive in a given setting may facilitate using cCBT where it will be best received by patients.

Allow ample time for required institutional agreements

The timeline for receiving institutional approvals in partnering with an external vendor was longer than originally anticipated. We experienced delays in the process and were thus delayed in implementation. Therefore, it is critical to understand the range and scope of the institutional approval process for an intended setting and to plan for what may be an extended timeline. Although approval processes will differ by organization, we anticipate that most institutions will have processes related to vendor agreements, data sharing and hosting, procurement, HIPAA compliance and patient safety.

Invest in staff training to describe the program and set patient expectations

Only 44.4% of patients who started the program completed all 9 modules and this number was lower (22.2%) for those in the augmented-psychotherapy program. Although we did invest time in staff training and processes to discuss the program with patients, additional efforts up front may have better matched patients to the cCBT program. Research into the types of patients who find cCBT most acceptable will also support improved matching of patients to these types of programs.

Introduce cCBT to patients early in the treatment process

Patients who started directly into the cCBT-only program had much higher engagement than patients using it as an augment to traditional psychotherapy. These types of programs may be best utilized by patients seeking them out directly and/or when incorporated early in the psychotherapy treatment process.

Identify a clear means for technical support in advance

Patients encountered technical challenges and questions, many of which clinicians were unable to troubleshoot. Although technical challenges are likely to be encountered in any program of this type, it may be beneficial to clearly define the process for troubleshooting technical issues in advance. For example, it may be helpful to establish clarity about the technical assistance process with vendors early in the planning stages and have vendors train clinic personnel regarding common technical issues. Additionally, incorporation of digital navigators in settings that adopt computer-based psychotherapy programs may be an alternative and acceptable approach to supporting patient engagement with technology (18).

Allow clinicians to opt-in during early program implementation with planned expansion to other clinicians over time

Participating clinicians opted-in to the cCBT program and their satisfaction with cCBT clinical work was high. This opt-in framework is likely to facilitate successful early implementation of similar programs and help to build clinical champions who can enhance buy-in among other clinicians in subsequent expansion (19). In addition, the high satisfaction among clinicians in this program suggests these types of programs may be beneficial for provider satisfaction and wellbeing as they provide clinicians with additional tools to utilize with patients and provide variety in their daily work practices.

Consider customizability of online content during vendor selection

There was high variability in which modules patients found most useful. In addition, during the program patients sometimes identified a desire to address symptoms or challenges in areas for which there was not directly relevant online content available. Programs in which patients and clinicians can select and customize their online content based on a menu of clinical concepts may increase effectiveness and patient satisfaction.

Integration with research

This was a quality improvement project and thus our findings may not be generalizable to other settings or populations. However, this pilot implementation highlighted areas where research may be especially beneficial in guiding cCBT program development and implementation. For example, research has shown smaller treatment effects for cCBT in primary-care settings compared to non-primary care settings (10). However, we had very low uptake of the program in our specialty mental-health setting. Because patients in primary care and integrated health care settings may have a less specific vision of the type of treatment they wish to seek, they may have greater uptake of cCBT. Future study of tradeoffs between treatment effect and patient uptake in differing settings would be informative. In addition, research has found that acceptability of cCBT differed by patient age (20). Future research that informs how to best triage and/or match patients to cCBT will help to optimize its application.

Consistent with research findings (6), patients and clinicians in our program found the live support component to be a valuable component of the program. That is, the relative acceptability and effectiveness of the program may have been due to its use as an enhancement to clinician-directed care rather than a replacement for it (21). Patients reported the live sessions were useful in holding themselves accountable for module completion and application of concepts to their own lives and experiences. Clinicians found it beneficial to offer additional psychoeducation and strategies beyond what was included in the modules and perceived that the additional material was well-received by patients. Although prior research clearly shows that live-support is beneficial, prior work has also included a wide range of variability in terms of how the support is provided (e.g., telehealth, telephonic, asynchronous email/messaging) and the credentials of those providing it (e.g., nurses, behavioral technicians, coaches, licensed clinicians, digital navigators) (6). Thus, in designing our program, we found there was not a clear consensus on how to best design and offer live support. Research that evaluates differences in live-support methods in terms of patient engagement and cost-effectiveness would be beneficial in further guiding development of computer-assisted psychotherapy programs.

Our finding that patients and clinicians were frustrated by technical issues during the program aligns with the literature on the importance of considering and supporting patient and clinician digital literacy when providing technology-based care (22-24). There is significant variation in patient and clinician comfort and knowledge related to using various technology-based mental health apps, platforms, and programs. A growing body of literature demonstrates support for inclusion of well-trained digital navigators in mental health care who can provide technical assistance to patients and clinicians, evaluate and recommend tools for incorporation into clinical settings, and review and discuss data collected by digital technologies to monitor symptoms and promote patient engagement (18,25,26). Lastly, based on the positive feedback from a student therapist working clinically in our program, cCBT and similarly structured programs may be useful tools for helping clinicians to expand or deepen expertise in specific approaches. We believe this is a promising area for research and future applications of computer-assisted psychotherapies.

Strengths and limitations

The primary strength of this project was its implementation in a real-world clinical setting that allowed us to assess uptake and engagement, acceptability, and impact on anxiety and depressive symptoms in an environment with few constraints and where patients had treatment choices other than cCBT. We demonstrated that cCBT was moderately acceptable for a subset of patients in a real-world setting and associated with reduction in depressive symptoms among those who participated in the program, including patients who would not have received care otherwise in our setting. Experiences with this program also highlight strategies and research that may enable more successful implementation of similar programs in the future.

There are important limitations to our evaluation of the cCBT program. There was very low program uptake by patients in our setting and reasons patients opted not to participate in the program were not tracked. Ultimately, findings regarding program experiences are derived from only ~3% of the patients in our clinic setting and we cannot speak definitively to the specific reasons patients opted not to participate. In addition, in the augmented-cCBT pathway, clinicians selectively offered the program to patients they felt would benefit from it. Thus, the uptake rate for the augmented pathway may be biased as it is unknown how many established patients who were not offered the program would have chosen to participate. We are also unable to speak to considerations related to long-term financial sustainability of a program like this due to the special funding streams for our program, which allowed us to offer the cCBT-only program at no cost to patients. In considering potential for a longer-term financial model, the live support appointments would likely have fit relatively directly into existing payment and payor infrastructures and been eligible for charges and/or applicable insurance reimbursement. However, because subscriptions or fees associated with online mental health programs are not typically covered by insurance (27), the cost of the online modules would likely have required the clinic to charge a fee directly to patients or to include the cost of the program in clinic operating expenses. We received mixed feedback from patients in our program regarding their willingness to pay for such content so it is unclear if a funding model that charged patients a fee for access to computer-assisted content would be viable. If online programs are able to demonstrate cost-savings to payors, insurance and employee benefit programs may increasingly cover similar programs (28), which would improve financially sustainability of this type of program.

Another important limitation to our program was the relative lack of diversity among our patient population. Our clinic serves a patient population where a large majority are privately-insured, report English as their first language and identify as White/Caucasian and not-Hispanic or Latino/a. Thus, it is unclear how the program would be utilized and accepted among a more diverse group of patients. Broader concerns related to diversity, equity and inclusion have been previously noted with regard to implementing technological innovations in healthcare (28,29). Those looking to implement similar programs in the future may benefit from proactively considering ways to make technology accessible to all patients such as providing internet connected tablets for use in accessing online content in the clinic and selecting vendors and content that depict a wide range of perceived demographic groups and lived experiences. Regarding the content of the cCBT programs themselves, we believe there is strong potential for the programs to foster and enhance inclusivity by offering content in multiple languages and with diverse vignette depictions, examples and avatars. However, this level of inclusiveness will require thoughtful and extensive industry commitment to developing diverse content.

Conclusions

Despite the low uptake in our setting, we believe there is good potential for this type of program to increase access to psychotherapy services based on the generally positive experiences of clinicians and patients who did participate in the program. cCBT programs may have greater uptake in settings where patients have not yet considered the specific type of treatment they prefer and/or where they remain open to a variety of treatments (e.g., primary care, employee assistance programs). We also anticipate that larger investment up front in describing potential benefits of the program would enhance uptake. As an alternative, a patient “opt-out” approach where all new patients are initially enrolled into a cCBT pathway while awaiting other services could help nudge patients toward greater willingness to try this type of program. Ultimately, we hope lessons learned in our program and future research in the area will inform successful implementations of subsequent cCBT programs designed to enhance access to high-quality mental health care.

Acknowledgments

We gratefully acknowledge Lidia Feseha for her support in discussing this program with patients and tracking data regarding patient uptake. We also wish to thank Polly Davis and Becky Alberti-Powell for their project management support.

Footnote

Peer Review File: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-24-22/prf

Funding: This work was supported by internal funding from the Chancellor’s Office at the University of Colorado-Anschutz Medical Campus.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-24-22/coif). M.M. serves as an unpaid editorial board member of mHealth from March 2023 to February 2025. D.S. and M.M. worked with MindStreet, Inc. to obtain reduced fees for the Good Days Ahead licenses. In return for the reduced fees, Mindstreet, Inc received the right to (I) advertise their partnership with our institution; (II) access data from our pilot program and publicize results; and (III) receive introductions to university and affiliate health plan personnel. This relationship did not impact reporting of this project. This project was funded in part by the University of Colorado-Anschutz Medical Campus Chancellor’s Office. A.D. has a contract to conduct a pilot implementation of BabyScripts, a digital based platform for postpartum mental health education and symptom monitoring; she has received grant funding from the Little Giraffe Foundation, Newborn Hope Foundation, Zoma Foundation, and HealthRhythms; she has received speaker honoraria from the National Perinatal Association, Cornell School of Medicine, and Dell Medical School and is an unpaid member of the American Psychological Association CPT Advisory Council. J.S. is Chief Medical Officer at Access Care Services which provides telemental health services in Colorado and Alaska; he has received royalties/honorarium from American Psychiatric Association, Springer Press and Talkiatry for presenting at Grand Rounds and serves as a member of the Board of Regents for the American College of Psychiatrists. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The project was deemed quality improvement and exempt from review by the Colorado Multiple Institutional Review Board.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Cuijpers P, Quero S, Noma H, et al. Psychotherapies for depression: a network meta-analysis covering efficacy, acceptability and long-term outcomes of all main treatment types. World Psychiatry 2021;20:283-93. [Crossref] [PubMed]
2. McHugh RK, Whitton SW, Peckham AD, et al. Patient preference for psychological vs pharmacologic treatment of psychiatric disorders: a meta-analytic review. J Clin Psychiatry 2013;74:595-602. [Crossref] [PubMed]
3. Areán PA, Renn BN, Ratzliff A. Making Psychotherapy Available in the United States: Implementation Challenges and Solutions. Psychiatr Serv 2021;72:222-4. [Crossref] [PubMed]
4. Cavanagh K, Herbeck Belnap B, Rothenberger SD, et al. My care manager, my computer therapy and me: The relationship triangle in computerized cognitive behavioural therapy. Internet Interv 2018;11:11-9. [Crossref] [PubMed]
5. Hermes ED, Merrel J, Clayton A, et al. Computer-based self-help therapy: A qualitative analysis of attrition. Health Informatics J 2019;25:41-50. [Crossref] [PubMed]
6. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther 2009;38:196-205. [Crossref] [PubMed]
7. Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev 2012;32:329-42. [Crossref] [PubMed]
8. Shafran R, Clark DM, Fairburn CG, et al. Mind the gap: Improving the dissemination of CBT. Behav Res Ther 2009;47:902-9. [Crossref] [PubMed]
9. Sharry J, Davidson R, McLoughlin O, et al. A service-based evaluation of a therapist-supported online cognitive behavioral therapy program for depression. J Med Internet Res 2013;15:e121. [Crossref] [PubMed]
10. Wright JH, Owen JJ, Richards D, et al. Computer-Assisted Cognitive-Behavior Therapy for Depression: A Systematic Review and Meta-Analysis. J Clin Psychiatry 2019;80:18r12188.
11. Wright JH, McCray LW, Eells TD, et al. Computer-Assisted Cognitive-Behavior Therapy in Medical Care Settings. Curr Psychiatry Rep 2018;20:92. [Crossref] [PubMed]
12. Hofmann SG, Asnaani A, Vonk IJ, et al. The Efficacy of Cognitive Behavioral Therapy: A Review of Meta-analyses. Cognit Ther Res 2012;36:427-40. [Crossref] [PubMed]
13. Adelman CB, Panza KE, Bartley CA, et al. A meta-analysis of computerized cognitive-behavioral therapy for the treatment of DSM-5 anxiety disorders. J Clin Psychiatry 2014;75:e695-704. [Crossref] [PubMed]
14. Patel S, Akhtar A, Malins S, et al. The Acceptability and Usability of Digital Health Interventions for Adults With Depression, Anxiety, and Somatoform Disorders: Qualitative Systematic Review and Meta-Synthesis. J Med Internet Res 2020;22:e16228. [Crossref] [PubMed]
15. Lorenzo-Luaces L, Wasil A, Kacmarek CN, et al. Race and Socioeconomic Status as Predictors of Willingness to Use Digital Mental Health Interventions or One-On-One Psychotherapy: National Survey Study. JMIR Form Res 2024;8:e49780. [Crossref] [PubMed]
16. Patel SR, Wheaton MG, Andersson E, et al. Acceptability, Feasibility, and Effectiveness of Internet-Based Cognitive-Behavioral Therapy for Obsessive-Compulsive Disorder in New York. Behav Ther 2018;49:631-41. [Crossref] [PubMed]
17. Becker EM, Jensen-Doss A. Computer-assisted therapies: examination of therapist-level barriers to their use. Behav Ther 2013;44:614-24. [Crossref] [PubMed]
18. Perret S, Alon N, Carpenter-Song E, et al. Standardising the role of a digital navigator in behavioural health: a systematic review. Lancet Digit Health 2023;5:e925-32. [Crossref] [PubMed]
19. Santos WJ, Graham ID, Lalonde M, et al. The effectiveness of champions in implementing innovations in health care: a systematic review. Implement Sci Commun 2022;3:80. [Crossref] [PubMed]
20. Donker T, Batterham PJ, Warmerdam L, et al. Predictors and moderators of response to internet-delivered Interpersonal Psychotherapy and Cognitive Behavior Therapy for depression. J Affect Disord 2013;151:343-51. [Crossref] [PubMed]
21. Bond RR, Mulvenna MD, Potts C, et al. Digital transformation of mental health services. Npj Ment Health Res 2023;2:13. [Crossref] [PubMed]
22. Connolly SL, Kuhn E, Possemato K, et al. Digital Clinics and Mobile Technology Implementation for Mental Health Care. Curr Psychiatry Rep 2021;23:38. [Crossref] [PubMed]
23. Kopelovich SL, Buck BE, Tauscher J, et al. Developing the Workforce of the Digital Future: mHealth Competency and Fidelity Measurement in Community-Based Care. J Technol Behav Sci 2024;9:35-45. [Crossref] [PubMed]
24. Torous J, Erica C, Keris M. Equitable and Informed Digital Mental Health: Skills, Evaluation, and Integration of Apps into Care. Technology, Mind & Society Conference Proceedings; 2021.
25. Alon N, Perret S, Cohen A, et al. Digital Navigator Training to Increase Access to Mental Health Care in Community-Based Organizations. Psychiatr Serv 2024;75:608-11. [Crossref] [PubMed]
26. Wisniewski H, Torous J. Digital navigators to implement smartphone and digital tools in care. Acta Psychiatr Scand 2020;141:350-5. [Crossref] [PubMed]
27. Spurgeon JA, Wright JH. Computer-assisted cognitive-behavioral therapy. Curr Psychiatry Rep 2010;12:547-52. [Crossref] [PubMed]
28. Insel T. Digital mental health care: five lessons from Act 1 and a preview of Acts 2-5. NPJ Digit Med 2023;6:9. [Crossref] [PubMed]
29. Weith J, Fondacaro K, Khin PP. Practitioners’ Perspectives on Barriers and Benefits of Telemental Health Services: The Unique Impact of COVID-19 on Resettled U.S. Refugees and Asylees. Community Mental Health Journal 2023;59:609-21. [Crossref] [PubMed]