Lessons learned launching an engaging patient portal: an exploratory case study examining the launch and integration of the patient portal into the continuum of care for substance use treatment

There has been acknowledgement for many years within the field of substance use treatment that substance use disorders are a chronic condition, requiring a continuum of care and support rather than perceptions of it as an acute condition utilizing more time-limited services. The idea of substance use disorders as a chronic illness has been established since the 1950s and has been supported by important research, including a study published by McLellan and colleagues (1), which demonstrated that return to use rates (sometimes referred to relapse rates) are similar to rates of the return of symptoms for other chronic disorders such as hypertension, diabetes, and asthma. As is standard with the treatment of chronic conditions, long-term treatment engagement is therefore suggested to be important for helping to improve outcomes and overall quality of life for individuals with these diagnoses.

Although the idea of longer treatment engagement is often supported by clinicians, there are many barriers with patients that can prevent this from occurring (Campbell et al. 2012) (2). These barriers may include geographical access, financial restrictions, and other life factors. As a result, substance use treatment providers are exploring innovative ways to provide ongoing care and support. One of the avenues explored has been the use of patient portals as a means for augmenting treatment and extending the reach of clinical and supportive services. A patient portal is a secure online website that gives patients convenient, 24-hour access to personal health information from anywhere with an Internet Connection (HealthIT.gov, 2017) (3). Patient portals are utilized for many purposes, including secure communication/setting appointments with providers, reviewing lab/test results, reminders for preventative care, health education and more. This article reviews the literature on factors associated with the use of patient portals and then provides a case study review of the actual process engaged in by the Hazelden Betty Ford Foundation in the launch of the MyRecoveryCompass patient portal and the My Ongoing Recovery Experience (MORE)^® comprehensive substance use disorder/recovery support program that is integrated within the patient portal.

Treatment engagement

The authors of this article spent time in preparation conducting an unofficial poll of practicing clinicians (n=10) from a variety of treatment settings, asking whether treatment length was associated with treatment outcomes. Nine of the ten clinicians asked responded that longer treatment engagement was related to better outcomes; however, when asked a follow-up question about where there was research supporting that, only two clinicians were able to reference having ever read research that supports that belief first hand. Of the research that does exist supporting longer lengths of treatment being associated with better outcomes, only a few of those were conducted outside of college counseling settings from U.S. universities (Falkenstorm et al. 2016) (4). As such, there has historically been a belief within the behavioral health field suggesting that longer engagement supports better outcomes without a wider range of research supporting this conclusion for settings beyond university counseling centers. Studies that were conducted more specifically aimed at substance use treatment engagement often include a review of substance use disorder after-care programs, also known as recovery support, in contrast to just classic clinical levels of substance use disorder care; however, these studies often lack a control group comparison and so it is difficult to differentiate the effects of more traditional substance use disorder treatment engagement and longer-term engagement with substance use disorder after-care programming (Proctor & Herschman, 2014) (5). When considering substance use treatment, DuPont and Humphreys (6) suggested that previously reported levels of continuous sobriety reported by health care professionals (81%) was likely supported by participation in long-term recovery support programming.

Falkenström and colleagues (4) tested this hypothesis in outpatient settings and found that, consistent with established research, their results suggested that longer engagement in treatment resulted in better outcomes. Although their study was limited to 12 sessions, which is shorter-length treatment than might otherwise be suggested for long-term engagement (e.g., services that include less than 16 sessions are often referred to as brief interventions in the research literature), the implications for overall provisions of care are important. First, they suggested that longer engagement in treatment (when clearly established to be a longer-term approach to care) resulted in slower but better outcomes. In other words, when clinicians and patients know there is more time to achieve treatment goals, there are often longer lengths of time needed to make as much progress, but the progress in treating symptoms is significantly better than shorter-length treatment approaches. Second, there is a natural lull in the middle of long-term treatment. Specifically, there were greater rates of change for individuals in long-term treatment plans at treatment onset and prior to termination than in the middle of treatment. This treatment progress lull was common in their research participants and routinely passed with time. The implications of these findings support that in-process measures of treatment progress should be interpreted with caution so that this normalized treatment lull is not confused with reduced efficacy of treatment.

Replicating results from other researchers, Erekson et al. (7) found that more frequent treatment engagement was associated with faster recovery rates (Cohen’s ʃ 2 =0.07 with a small to medium effect size) than lower frequency interventions. Erekson et al. (7) suggested that frequency of engagement should be considered when implementing treatment planning. When increased session frequency is not an option, other technologies might offer alternatives to help increase patient engagement between sessions.

When considering treatment engagement, numerous factors have been identified as creating barriers to patients engaging in long-term care. These factors include, but are not limited to, financial limitations, life responsibilities, geographic proximity to treatment, and participant motivations (Rios, Kazemi, & Peterson, 2018) (8). When considering these barriers, Rios, Kazemi, and Peterson (8) suggested that providers consider treatment innovations to increase access, including strongly asserting that technology can be utilized as a means for closing the treatment gap, especially for patients in rural areas where access to other services are limited. This emphasis on creating ease of access for patients is important for the provision for care, especially when the goal is to remove engagement barriers that might otherwise exist to effect population subsets (e.g., rural populations, low-income populations, etc.).

Similarly, Santa Ana and colleagues (9) suggested that access to care is the primary reason for providers to consider seriously the provision of services through technology options. They recommended that any technology utilized should be easy to use and accessible by the patients it is designed to support. Further recommendations include that technology may best work as a platform for treatment (e.g., the platform for delivery such as with virtual visits or adjunctive content to support other clinical interventions) rather than a substitute (e.g., use of apps replacing clinical providers). In response to these and other similar conclusions, treatment providers have shifted to the use of technology as a platform for providing or augmenting treatment for patients.

Specifically, within the provision of substance use treatment, there are certain factors that further present risk for patient engagement and access that can be addressed, at least in part, by the integration of technology into how substance use treatment services are provided. Barker and colleagues (10) identified patient characteristics that were more likely to drop out prematurely from residential substance use treatment. Individuals with these circumstances may be at higher risk of attrition and therefore technology-based interventions designed to help provide substance use treatment engagement as well as support post engagement can be essential for the well-being of these individuals. Characteristics identified by Barker and her fellow researchers associated with higher risks of attrition included: (I) younger patient populations; (II) having less than a high school education; (III) having an unstable living situation; (IV) having had experienced either great use of substances (not including alcohol) or reduced use of alcohol in the month prior to admission; (V) needle use; and (VI) having been recommended to have longer lengths of stay. With a greater risk for attrition, it becomes essential that patients with these circumstances be engaged quickly and effectively with the means to provide on-going support in the case that there is otherwise substance use treatment attrition.

The use of technology in treatment

Rios, Kazemi, and Peterson (8) suggested that any service provider that is interested in providing support to patients via technology should consider the following four factors prior to launching any new initiatives: (I) what technology will be used; (II) what hardware would be required to access it; (III) what software would be used to facilitate it; and (IV) how is security managed. The use of technology to assist in the provision of clinical treatment has resulted in several new frontiers, including the creation of apps, telehealth options, and the use of Electronic Health Record (EHR)-based patient portals as avenues of care; however, each approach has both its benefits and potential obstacles.

Bush et al. (11) reviewed the use of apps for psychological interventions. They shared that there is currently no standardized process for reviewing the efficacy of apps, and suggested that due diligence is required on behalf of the clinician who decides to integrate these technologies to verify the efficacy and authenticity of the content. In other words, clinicians are responsible for verifying the reputation and validity of the approaches used if they are going to recommend these technologies as an adjunctive part of their patient care. This recommendation suggests that there is a strong need within the treatment community for high-quality, web-based content that comes from reputable sources, and that practicing clinicians should do their best to ensure that they use these technologies when providing care to patients.

Clinicians, especially prior to using technology, often reported perceiving telehealth and other technology-based approaches to delivering care as a threat to their own clinical autonomy, which is a belief that has created barriers to the use of technology. Campbell and colleagues (2) reported on their attempts to launch a web-based version of a supportive psychological instrument for the treatment of substance use disorders. Although their review was published early in their launch phase, the authors reported both successes and early lessons learned that would be helpful to providers considering a similar expansion into web-based supports for patients. The preliminary results of their pilot were suggesting positive outcomes, and the major concern identified was the initial barrier in obtaining clinician buy-in. The authors reported that many clinicians perceived web-based technologies as a potential threat to their clinical autonomy or that technology was potentially going to replace their role. Campbell and her team suggested that this barrier was best overcome by providing significant front-end training as well as providing clinicians with access to review the content provided to patients. Further success was gained when they presented technology as a “clinical extender” (2) that provided the clinician with increased means for providing support and services to the patient. This perception greatly increased clinician buy-in by positioning patient engagement with technology as something guided by the clinician rather than supporting the perception that technology was a replacement for clinical interventions.

More recent research reported on by Heath (12) stated that clinicians were often reluctant to adopt new technologies until they have reviewed what other clinicians have experienced with integrating those technologies into care. In addition to barriers with clinicians promoting the use of these technologies, including patient portals, Heath indicated that patients themselves may opt out of using portals if they are unaware of the features they could access through this as well as if the portals themselves are not designed in an easy-to-use manner. Health noted that patients may not have a lot of health literacy and therefore may not be aware fully of the potential benefits to their care that accessing these technologies may have on their overall patient experience. Between potential reluctance on behalf of the patient and non-promotion of use on behalf of the clinicians, Heath shared that it is estimated that close to 90% of providers offer a patient portal, but only roughly a third of patients are actually accessing the portal. This suggests that there are still major barriers that need to be overcome in order for these technologies to be able to support the patient experience fully.

One of the barriers to clinician buy in, as noted by Campbell and colleagues (2) was that there are concerns about the efficacy and safety of web-based interventions. These concerns have been the focus of many articles suggesting best practices for this emerging treatment modality. Edwards-Stewart and colleagues (13) reported on ethical and legal considerations for when providers utilize mobile applications. They shared that mobile technology is promising and research has found the use of adjunctive mobile technology to help reduce a range of presenting problems, including substance use disorder symptoms. However, they also noted that there is a current lack of clear guidelines for the use of technology when providing services and that there are some real barriers to conducting traditional efficacy studies for content-based technologies. One barrier identified specifically was related to the rate of technology advancements and updates. When referencing back to Rios, Kazemi, and Peterson (8), it becomes clear that software and hardware can be updated in a very expedient fashion. It is not uncommon for web-browsers to go through rapid version updates, especially when there has been a potential security risk identified, and the advancement of hardware, including advancements in upgraded phone versions, computers, and tablets, are also released frequently. As such, Edwards-Stewarts and colleagues (13) warned that traditional approaches of using a variety of research designs to test for efficacy would not work as well for testing these applications. For example, a longitudinal study of efficacy may span across several updates in software or technology, any of which may change the user experience in a variety of ways.

In order to address these concerns, Edwards-Stewart and colleagues (13) suggested that service providers rely on the American Psychological Association ethical principles of Beneficence and Non-maleficence, in that providers do what they can to ensure they minimize the risk of any potential harm. This requires review, oversight, and a critical evaluation of provision of any technology that can be utilized as an adjunctive and technology-based aspect of care (as well as an overall standard of practice for all types of care provided by clinicians). Additional considerations are suggested by Edwards-Stewart and colleagues that include urging that there are adequate informed-consent procedures to let the users of the technology know the potential benefits, risks, and scope of care provided with these technologies.

EHR patient portals

The use of patient portals as a means to help engage patients is not necessarily a recent innovation. As noted by Oss [2019], many providers started offering patient portals as a means to meet Meaningful Use guidelines. However, Oss [2019] reviewed the current state of patient portals and suggested that a recent publication by the U.S. Government Accountability Office suggested that although 88% of hospitals were offering patient portals, only roughly 15% of patients were actually using this technology. Oss explored the current research literature and suggested that many factors, including patient concern about the usefulness of the portal and timeliness of communications, were identified as reasons for why this technology was not better utilized. Oss argued for the potential for patient portals to be both useful and key factors in the patient experience, and suggested that providers find ways to make the use of these portals as an integral part of the treatment process in order to help boost usage and engagement. To encourage the best integration of patient portals into treatment, innovations and engaging content must be included.

Marrow and colleagues (14) made specific recommendations for how to create engaging content to share with patients on EHR patient portals. These recommendations include making content that provides high-level ideas that are easy to understand but that do not require the viewers to memorize information that is too specific. The authors noted that participants in their study, especially individuals from older age demographics, found that there was better comprehension for shorter videos and articles, especially when the intended goal was for the patient to leave with a general idea of what to do next rather than when the goal was to provide specific instructions. The authors also shared that patients reported increased feelings of negative emotions and perception of risk associated with test results posted online when they believed that the results of the test were related to high perceived risk. In other words, when a patient completed an assessment that might indicate the presence or absence of symptoms they were nervous about, then patients were more anxious and less hopeful reading about these results online than when they went over the results with a clinician in person. This is likely due to limitations in asking clarifying questions as well as potential contextual cues patients would receive from the manner in which the test results are shared in person versus the lack of context included when results are simply posted on the EHR.

The creation of content that is both useful and engaging can be an obstacle for providers looking to optimize their patient portals as adjunct means of care. Santa Ana and colleagues (9) recognized a lack of clear guidance in the use of technology to help augment treatment. As demonstrated with their own pilot study of launching adjunctive services through technology innovations, results were promising for effective treatment outcomes. With these promising outcomes in mind, Santa Ana and her fellow researchers provided guidelines for other individuals in the creation of innovative technology to support treatment. These recommendations included addressing patient motivation for using the technology, clinician engagement/support, and addressing the concern about how this technology will alert clinicians to risk. They also recommended that developers rely on subject matter experts who are also clinicians to develop the content. By following these and other guidelines outlined in research, providers can find pathways to creating meaningful content to best support patients in their recovery journey.

Case study

The Hazelden Betty Ford Foundation is a non-profit substance use treatment provider with seventeen locations throughout the US. The following is a case study outlining the process taken by the organization to optimize its patient portal, known as “MyRecoveryCompass” to host and support its My Ongoing Recovery Experience (MORE)^® platform as a means to increase patient engagement and to offer long-term support for patients with substance use disorders, including aftercare services and supports.

History

Hazelden Betty Ford created the MORE^® program in 2006 as a way to help provide substance use disorder patients with support extending past treatment by offering both a recovery coach and a collection of recovery support articles, activities, and videos. The content was created by in-house subject matter experts utilizing evidence-based practices such as Twelve Step Facilitation, Cognitive Behavioral Therapy, etc.…as the foundation. This content was placed into an online comprehensive support program. Activities and articles were designed to address tasks associated with living in recovery (e.g., activities identifying recovery supports in the community, and articles discussing how to select and engage with a sponsor). A version of this program was later modified and customized for the U.S. Navy (which was branded NAVY MORE^®). The MORE^® program was also the recipient of awards for content including the silver medal for eHealthcare Leadership Award for Best Care/Disease Management Site in 2007, as well as the White House Award Behavioral Health Patient Empowerment Challenge award (Cision PR Newswire, 2013) (15).

Although Hazelden Betty Ford had spent considerable resources developing this recovery specific content, it was not being utilized as well as expected by patients or clinicians. Obstacles included clinician support and patient awareness. Furthermore, feedback gathered over the course of use suggested that patient engagement could be improved through optimizations. When MORE ® was first designed, many of the articles and activities were separated into a sequential, timed curriculum. Patients using MORE^® were prevented from accessing new content until they had completed all of the tasks associated with each section of the comprehensive support system that included resources (e.g., articles and activities) and after a set amount of time had elapsed. These sections had been divided based on the milestones of recovery in the order in which the research had suggested people would transition through these topics. This research had been conducted through the Hazelden Butler Center for Research and complied results of several post-discharge studies examining self-reported needs of patients following the completion of the first year post-treatment (Owen, 2003). Based on results from this research, sections of content (also referred to as modules) were created and timed according to when patients had indicated that they experienced time-specific stressors related to the recovery process. Access to modules were divided into time-based curriculum, where modules became available at specific post-discharge intervals. This decision was based on a combination of the milestones of recovery data and the initial belief that if the entire curriculum was made available immediately then patients might not spend as much time using the resources and therefore not gain the full benefit this comprehensive support system had to offer. However, this model of delivery was changed after feedback obtained by users suggested that this sequential and timed curriculum design was actually a barrier to patients engaging in the topics that were important to them, when they wanted to engage in it. Based on this feedback, MORE^® was re-designed to allow for all content to be available to patients immediately.

Another re-design to the MORE^® program was based on feedback that patients were not engaging as often as expected in the video content. Although the videos were high production quality and meant to be informative, there was a trend supported by data suggesting that these videos were used infrequently by patients. An exploration of this situation resulted in the discovery that, although patients appreciated the content, the actual video lengths were a barrier to use. MORE^® content was then re-edited, and in some cases, re-created, to make sure that the average video length was between three and five minutes. Shortening the videos and removing sequential/timed access restrictions from the MORE^® program were completed in 2016.

In 2016, Hazelden Betty Ford launched a new EHR. The launch of this EHR provided an opportunity to utilize the EHR’s existing patient portal in a more engaging way. This was when MyRecoveryCompass was created. The primary driving aim behind the revamping the MORE^® program and putting it on this portal was to increase patient engagement over a longer period of time. Hazelden Betty Ford recognized that the substance use treatment landscape was shifting. Along with greater utilization rates of the supportive and comprehensive recovery programing by their patient populations came overall decreased lengths of stay in more traditional clinical levels of care. Technology based innovations were seen as the means for best ensuring that patients would be able to stay engaged with the organization regardless of length of stay and the geographical location that patients returned to after substance use treatment.

As noted earlier, engagement levels on patient portals are often low (Oss, 2019). In order to increase utilization, Hazelden Betty Ford loaded the MORE^® program onto the MyRecoveryCompass platform. By integrating the MORE^® program with the patient portal, patients now had a single log-in where they could accomplish the following objectives: (I) communicate with providers; (II) complete online assessments; (III) access specific treatment records; (IV) access personalized treatment plans, and (V) access the MORE^® program, which included articles, activities, and videos. The launch of this combined patient portal was ambitious and not without challenges.

There were two major obstacles to utilization of the MyRecoveryCompass patient portal: (I) patient utilization of technology, and (II) clinician buy-in. Strategic analyses were conducted to best understand these obstacles as well as to design initiatives to increase engagement and utilization. The first discovery process explored patient utilization. It was discovered early on that patients were being informed of the patient portal toward the end of treatment, often just prior to discharge. This analysis found that many patients saw this as simply a way to get post-treatment information and not integrated into their treatment experience. In addition, with little experience and practice with the portal during treatment, the patients had lower competency and comfort with using it after they left. As such, few patients enrolled to gain access to the patient portal, and fewer patients used it.

Clinician buy-in was another major obstacle to patient utilization. A discovery process identified the four following obstacles to clinician support: (I) not knowing how patients register; (II) worry about content infringing on clinical autonomy; (III) lack of clinician support; and (IV) lack of familiarity with MORE^® content. In regards to worry about the content infringing on clinician autonomy, some clinicians reported concerns that portal content would be perceived by either patients or supervisors as replacing the need for clinicians. This fear was soon realized to be related directly to a lack of familiarity with the content of the MORE^® program. Due to technology limitations, clinicians were not able to be provided access and were unable to preview the MORE^® program. Once this was discovered to be a significant barrier, steps were taken to make MORE^® content available for clinician review, which greatly changed the levels of clinician support and utilization of MORE^®.

To best address the organizational aims of using MyRecoveryCompass, a multi-phase project initiative was developed. This included three phases: (I) increase patient enrollment; (II) increase patient usage; (III) increase clinician usage of information gathered through the portal. The first phase was an aim to increase patient enrollment in the MyRecoveryCompass patient portal. A strategic goal was set in 2017 to have at least 90% of all patients registered into the patient portal within the first 7 days of enrolling in treatment with Hazelden Betty Ford. This goal was designed to increase patient registration at the front end of substance use treatment rather than having registration be sought toward the end of treatment. The belief was that by having patients register early, then patients would be more likely to use the portal during substance use treatment, increasing their comfort, competency and familiarity with the program, and therefore, continue to use it as they transitioned out of treatment. With clinician training and technology support, Hazelden Betty Ford was able to achieve this goal by the end of 2017, and has maintained organization-wide enrollment rates above 90% throughout 2018 and into 2019.

The second phase of this initiative was launched in 2018 and focused on trying to increase patient use of the patient portal. The strategic goal was to have clinicians in the intensive outpatient programs add treatment plan items related to the use of the MORE^® program. The reason this goal was added is that it would require clinicians to discuss the MORE^® program with patients as well as provide patients a specific goal related to substance use treatment that required them to gain exposure to the patient portal and the MORE^® content. The hypothesis guiding this initiative is that if patients have a reason to access and use the patient portal during substance use treatment, then they will be more likely to continue to use the patient portal post-discharge to support their long-term recovery.

In launching phase two, Hazelden Betty Ford found that the barriers to achieving the goal were similar to those identified in the literature, in that clinician buy-in was a major obstacle to success. Some clinicians worried that including a treatment plan item related to accessing the patient portal was removing their clinical autonomy. This concern was not overcome until clinicians were able to review the content first hand and realized that there was a whole program of potential activities, psychoeducation articles, workbooks, and other related content available for them to assign to their patients and that the selection process of what to assign required the clinician’s clinical judgment. Once this was understood, the clinicians shifted from viewing the patient portal as a potential replacement of their clinical skills to a means of supporting their clinical directions and care for their patients. With this new approach, the project management team was able to get all of the Hazelden Betty Ford Foundation substance use disorder outpatient programs to achieve the goal of 70% or more patients with MORE^®-related content on treatment plans. Furthermore, every site achieved this goal within six weeks of launch.

Phase three, which is currently in process, moves to further increase the utility of the patient portal by making it the platform through which in-treatment measures are delivered to the Hazelden Betty Ford patient population to provide measurement-based care. These assessments help to ensure that progress is measured throughout the course of substance use treatment rather than relying on more standardized post-treatment follow-up reports/outcome data. The goal of using the patient portal in this manner is to help identify critical factors related to patient progress in a manner unique to each patient during substance use treatment and therefore allow the clinician to best customize interventions in a responsive manner. A more long-term goal is to use this data to determine ways to be more predictive about what treatment interventions work best for a specific profile of patient.

Summary

Innovative approaches to best achieve clinician buy-in and support became a critical factor that allowed for the successful launch of the MORE^® program and MyRecoveryCompass patient portal; however, innovation often comes with some obstacles. The launch of the Hazelden Betty Ford patient portal, as outlined in this article, faced many of the same obstacles and opportunities as outlined in the current research literature. For example, consistent with the findings of Oss [2019] related to patient portals in general, only a small number of patients were using the Hazelden Betty Ford patient portal. In addition Hazelden Betty Ford found that, consistent with Falkenström and colleague’s (4) findings many clinicians think first about higher levels of treatment rather than just overall engagement as keys to long-term success. In order to help address this, Hazelden Betty Ford implemented strategic goals to help increase utilization of the patient portal. Finally, consistent with Marrow and colleagues (14), Hazelden Betty Ford found that shorter and more engaging content was linked to higher usage rates. Future initiatives and research will be designed to help demonstrate how this has affected the course and quality of care received by Hazelden Betty Ford patients.

Additionally, future-features of MyRecoveryCompass will pave the way for increased use of telehealth services in the substance use treatment field. As substance use treatment continues to become a blend of in-person and virtual experiences, a robust portal with innovative features will be a critical component for success with patients that may not physically be present at any in-person facility. An example of future technologies that could be facilitated by an innovative portal includes the ability to upload and analyze data from wearables (measuring such things as heart rate, sleep patterns and even evidence of substance use through the skin) resulting in alerts to clinicians and automated feedback loops.

Acknowledgments

None.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

References

1. McLellan AT, Lewis DC, O'Brien CP, et al. Drug dependence, a chronic medical illness: implications for treatment, insurance, and outcomes evaluation. JAMA 2000;284:1689-95. [Crossref] [PubMed]
2. Campbell AN, Miele GM, Nunes EV, et al. Web-based, Psychosocial Treatment for Substance Use Disorders in Community Treatment Settings. Psychol Serv 2012;9:212-4. [Crossref] [PubMed]
3. The Office of the National Coordinator for Health Information Technology (ONC) “What is a Patient Portal?” HealthIT.gov 29 Sept. 2017. Available online: www.healthit.gov/faq/what-patient-portal
4. Falkenström F, Josefsson A, Berggren T, et al. How much therapy is enough? Comparing dose-effect and good-enough models in two different settings. Psychotherapy (Chic) 2016;53:130-9. [Crossref] [PubMed]
5. Proctor SL, Herschman PL. The continuing care model of substance use treatment: what works, and when is "enough," "enough? Psychiatry J 2014;2014:692423. [Crossref] [PubMed]
6. DuPont RL, Humphreys K. A new paradigm for long-term recovery. Subst Abus 2011;32:1-6. [Crossref] [PubMed]
7. Erekson DM, Lambert MJ, Eggett DL. The relationship between session frequency and psychotherapy outcome in a naturalistic setting. J Consult Clin Psychol 2015;83:1097-107. [Crossref] [PubMed]
8. Rios D, Kazemi E, Peterson SM. Best practice and considerations for effective service provisions via remote technology. Behav Anal (Wash D C) 2018;18:277-87. [Crossref]
9. Santa Ana EJ, Stallings DL, Rounsaville BJ, et al. Development of an in-home telehealth program for outpatient veterans with substance use disorders. Psychol Serv 2013;10:304-14. [Crossref] [PubMed]
10. Baker DE, Edmonds KA, Calvert ML, et al. Predicting attrition in long-term residential substance use disorder treatment: A modifiable risk factors perspective. Psychol Serv 2019. [Epub ahead of print]. [Crossref] [PubMed]
11. Bush NE, Armstrong CM, Hoyt TV. Smartphone apps for psychological health: A brief state of the science review. Psychol Serv 2019;16:188-95. [Crossref] [PubMed]
12. Heath S. (2018). What are the top barriers to patient portal adoption, use? Patient Data Access News. Available online: https://patientengagementhit.com/news/what-are-the-top-barriers-to-patient-portal-adoption-use
13. Edwards-Stewart A, Alexander C, Armstrong CM, et al. Mobile applications for client use: Ethical and legal considerations. Psychol Serv 2019;16:281-5. [Crossref] [PubMed]
14. Morrow D, Azevedo RFL, Garcia-Retamero R, et al. Contextualizing numeric clinical test results for gist comprehension: Implications for EHR patient portals. J Exp Psychol Appl 2019;25:41-61. [Crossref] [PubMed]
15. CISION PR Newswire (Sept 16, 2013). Hazelden mobile app wins White House award. PR Newswire. Available online: https://www.prnewswire.com/news-releases/hazelden-mobile-app-wins-white-house-award-223948881.html