Usability testing of an electronic health application for patient activation on weight management

Introduction

Obesity is a leading public health concern in the United States (1). Obese individuals are at increased risk for hypertension, diabetes, cardiovascular disease, cancer, as well as all cause increased mortality (2,3). Unfortunately, prior studies found that physicians are not adequately identifying and intervening with overweight and obese patients (4). There are several reasons for this including, but not limited to, lack of time, training and reimbursement along with a feeling that the responsibility does not land on the primary care provider (5,6). Similarly, patients also may fail to discuss this concern with their providers because of discomfort, or lack of awareness, knowledge about, and confidence in making the necessary behavioral changes (7-9).

Improving patient activation is one technique that can be used to empower patients to make healthy behavioral changes (10,11). Patient activation is defined as an individual’s knowledge, skill, and confidence in managing their own health (12,13). Patient activation promotes self-directed behavioral changes through a collaborative patient-provider environment centered on improving the patient’s health. This concept also dovetails with current interest in the patient-centered medical home model, where patients play a large role in care activation during clinic visits, including discussions related to lifestyle changes. Overall patient-centered practice models serve to endorse patient-provider collaborative interactions and create environments that promote patient activation.

A variety of methods have been used to enhance patient activation (14,15). One promising but understudied method for enhancing patient activation is the use of technology in the primary care setting. Prior research has shown that technology can be beneficial for patient education in the primary care setting. Technology has been used for patient education in the inpatient (16,17), outpatient (18-20), and surgical setting (20,21). While there is a plethora of research supporting the use of technology for patient education across a variety of clinical encounters, there is a paucity of research on the use of technology to enhance patient activation.

Importantly, many studies have examined the effects of technology on patient satisfaction, improvement of knowledge, and changes in clinical decision making (16-22). Previous work has found that patients are satisfied with the use of technology in the clinical setting for education (16-18) and a limited number of studies have shown that there is an improvement in patient knowledge (16,20,22). Taken together, these findings suggest that technology use can be an effective and satisfactory method of providing patient education.

The primary purpose of the current study was to test the usability of a patient activation application for use by patients during primary care wait times. Usability testing is a user-centered approach to assessing and improving positive patient experience with new technology. This study examined usability in waiting rooms because wait times provide a window of opportunity for a teachable moment with primary care patients. The electronic application (mWRAPPED) follows weight management guidelines (23) with the goal of supporting weight management through patient and physician empowerment of discussions. The current study describes usability testing in the clinical setting with patients and physicians through (I) observation, (II) usability questionnaires and (III) quantitative data on success rates, time on task, and error rates. We present the following article in accordance with the MDAR checklist (available at http://dx.doi.org/10.21037/mhealth-20-119).

Methods

Overview of usability testing methods: usability testing methods followed procedures identified on the usability.gov website (24), evaluating user experience through satisfaction and efficiency. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by Institutional Review Board of West Virginia University (WVU) (NO.: FWA00005078) and informed consent was taken from all the patients. This testing was done using a two-cycle approach followed by testing in the clinical setting. Each cycle involved task specific instructions with application edits after each cycle. The application was finalized after the second cycle of task-specific usability testing. This final version of the application was used in usability testing in the clinical setting.

The application

The tablet-based mWRAPPED application (app) was developed by the lead author (T Haggerty). The mWRAPPED application was developed with a patient-engaged approach incorporating key patient and provider input and incorporated accepted clinical practice guidelines for weight management (23). Since the application is intended for use in the clinical waiting room, there are minimal concerns over technology availability and connectivity issues. The application was designed to run on a variety of tablets which are fiscally available ($30 and up).

Patients input their height and weight in the first screen of the application. They then receive information about their body mass index (BMI) and a patient-friendly explanation of this number. Patients are then provided information about risk factors associated with their BMI, if overweight or obese. They are also provided with locally available nutrition services, exercise options, and surgical options. Patients can select items for which they wish to receive further information. Next, the patient is given encouragement and guidance on how to discuss weight management with their primary care clinician. Finally, the application provides patients a template for making a plan to address their weight with their provider. Patients are e-mailed a summary of the services they picked and their weight management plan.

Participants

A total of 40 adult primary care patients at the WVU Department of Family Medicine clinic took part in usability testing of this application. Usability participants were identified through the practice clinicians and dietitians. Patients were contacted through a phone call by the Principal Investigator to inform them about the study and gauge interest in participating. Clinical usability participants were approached after checking in for their clinic visit. They were consented and provided the application on a tablet in the waiting room.

Procedure

Usability testing: prior to usability testing, the researchers developed a script to allow all testing to remain uniform with each participant. Participants were scheduled to come to the WVU Department of Family Medicine clinic at a time convenient for them. Each session included one participant and two investigators. Informed consent procedures were completed with all participants prior to the study. The participants were told that the purpose of the testing was for application design and functionality and not to test the participant’s skills on using a tablet. A concurrent think aloud technique (25) was used to understand participants’ thoughts as they completed the testing. Participants completed three steps (I) providing demographic information, (II) completing an assessment of technical effectiveness and (III) completing post-testing surveys to assess system usability.

To test the technical effectiveness of the application, participants were provided with six tasks to complete. Each task tested a different portion of the application using hypothetical patient data (Table 1). Research staff recorded the amount of time it took participants to complete each task. A critical error was recorded if the participant was unable to complete a task. An example of a critical error would include the inability to see and press the button to proceed to a next step thereby not completing a task. We used the 10-item System Usability Scale (SUS) (26) to assess participants’ assessment of the overall usability of the application. The SUS has been found to be a quick and reliable scale measure of usability for the evaluation of electronic systems (26). Scores on the SUS range from 0 to 100, with any score above 68 considered above average and any score below 68 considered below average for usability.

Table 1 Demographics of Usability Test Participants
Full table

Each session lasted approximately 1 hour. Participants provided information on an Android tablet (Samsung Galaxy Tab A). Prior to testing, the two tablets were fully charged. The only links available on the device home screen were to the application being tested and the study surveys. All surveys were completed electronically through RedCap, a secure web application that is HIPAA compliant for completing surveys.

Consistent with usability testing guidelines (24), five participants initially completed the study procedures. Based on their results and participant feedback, the application was altered for improved usability. Study procedures were then completed by the remaining five participants using the revised application. Examples of improvements made following the first round of testing can be found in Figures 1,2.

Figure 1 Application flow before and after usability testing.

Figure 2 Before and after application updates for entering patient weights.

Clinical usability testing

Following the two-step usability testing procedure, we then provided a tablet with the application to a convenience sample of 30 Family Medicine patients prior to their clinic visit. The patients were offered the tablet to take into the waiting room with them and they kept the tablet during the visit. At checkout the patients returned the tablet to the research team and completed a demographics and SUS. Participants who did not complete the SUS scale were removed from the analysis (N=7).

Results

Table 1 illustrates the demographics for usability participants; 80% were female. The age range for all participants was between 18 and 65 years of age. Participants had an average BMI of 37.3 (within the obese category) with a range from 27.2 to 44.2. Half of the participants [5/10] rated their computer knowledge as high and only one participant rated her knowledge as low. Most participants [6/10] rated their comfort in accessing medical resources on technology as medium.

Technical effectiveness

All participants completed a majority of tasks (96%) in one attempt. Additionally, participants were able to complete the tasks in an average of 32.6 seconds. Throughout both rounds of testing, only one critical error by one participant occurred, and took place on task 3 (Table 2).

Table 2 Technical effectiveness testing tasks
Full table

A few issues were noted by the investigators during application testing. During the first testing cycle, all participants on task 1 tried to enter their height in feet and inches when the application asked for height in inches only. After task 1 (Table 2), the participants were given a height and weight to enter. One participant stated “Instead of doing mental math for height, just do feet and inches”. This feedback was incorporated into the application prior to the second round of testing.

Task 3 required that participants select locally available resource’s for nutrition, exercise, and bariatric surgery. This task was found to have the longest completion rate of 83.09 seconds and was the only task that resulted in a critical error. This task was the first task to ask participants to click on tabs within the application (Figure 1). One participant was unable to identify the tabs, which resulted in the critical error. Participants did note frustration with the use of tabs in the task 3. Participants provided suggestions for improving task 3, including “have the tabs more pronounced, maybe underlined or use a different color” and “other than fixing the tabs, I wouldn’t change anything else”. These tabs were changes to allow a streamlined process of completing the application (Figure 1).

System usability

User satisfaction was generally positive during the first round of testing with the initial five participants. These participants provided SUS scores from 62.5 to 95 with an average of 76.5. Only one participant rated the application below average with a score of 65. Comments from the participants helped the study team to better understand patient perspectives on the application and to either retain or modify components of the application. For instance, one participant stated that “After I learned how to do it, it was fine” and another mentioned that “patients would have more questions for the provider if this was used in the waiting room”.

After making edits to the application based on the first round of testing, the five participants in the second round gave SUS scores ranging from 70 to 87.5 with an average of 80.5. None of the participants ranked the application as below average on the SUS. Largely positive comments were noted by participants in the second round, including “I think it is user friendly and easy to navigate”. One participant was unsure on how to scroll to the email option and therefore had a critical error in task 3.

Participants found that entering their weight was cumbersome, since they had to start at one pound and scroll to increase the weight. Participants found that they were scrolling for a long period of time; this was reflected in an increase in task completion time from the first round of testing to the second round (Table 3) due to limiting the lower weight range to 80 pounds and using 5-pound increments. However, participants in round two also did not mention any difficulty in entering height into the system, as the system had been changed from to input height as feet and inches, in accordance with the suggestions from round one participants (Figure 2). Participants in round two focused more on the content of the application instead of the flow and made comments such as “make the community resource links work” and “I like that it explains what BMI is”.

Table 3 Technical effectiveness outcomes
Full table

Clinical usability

Table 4 shows the demographics for the 30 clinical usability testing participants. There was an equal number distribution of males and females with an age range of 18 to 74. These participants had a BMI range of 19 to 62; the majority of participants were overweight. Twenty-three of the 30 participants completed the SUS. The overall average SUS score was 77.9. The 18 participants that found the application usable provided an average SUS score of 83.19. Five participants found the application to be unusable with an SUS score of less than 68 with an average SUS score of 59.

Table 4 Demographics of clinical usability test participants
Full table

Discussion

This study used a patient-centered approach to test the usability of an electronic health application, mWRAPPED, to promote patient activation for weight management guidelines prior to their primary care visit. First, a two-cycle approach was used to develop the application. Next, the application was tested in the clinical setting. Results indicated that the current sample of adult primary care patients did find mWRAPPED to be usable.

This study has several unique aspects. First, it tested the usability of a patient activation application in the clinical setting. Second, this study utilized the patient waiting room to test the intervention. Third, this study is one of the few to test an electronic health application for weight management in the primary care setting.

This study has several limitations. Usability testing participants were mostly (80%) female, which may limit the generalizability of study findings to men. In addition, the oldest age of participants in the usability testing was 74 years of age with a majority of participants’ age under 55. As such, additional research is needed to understand the usability of this application with older adults. Clinical usability testing did include a more diverse group of participants. Finally, this testing was completed to assess usability alone and therefore no acceptability measures were collected. It should be noted that participants made positive comments related to the acceptability of using the application regularly in the clinic, though these comments were not systematically collected, as they were unrelated to the focus of the current study. Lastly, this application has not been tested for patient activation. Patient activation principals were used in the development of the application, however this will need to be studied. These limitations must be considered in the context of the larger purpose of this study. Results of the current study, while possibly limited in generalizability, have helped to improve the overall usability of the application which is an important first step before more large-scale testing. Next steps for the mWRAPPED application will include acceptability testing in the primary care setting prior to patients’ clinic visits.

Conclusions

Utilizing a two-cycle approach to usability testing, we were able to make an application that demonstrated initial clinical usability for primary care patients in a family medicine waiting room. Results of the current study and of future testing will help to support the use of this application as a novel approach to delivering guideline-based weight management information to patients. Additionally, information from this study will lead to the development of similar applications for treatment of other guideline-based treatment.

Acknowledgments

Funding: Research reported in this publication was supported by the National Institute of General Medical Sciences of the National Institutes of Health under Award Number 2U54GM104942-02. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Footnote

Reporting Checklist: The authors have completed the MDAR checklist. Available at http://dx.doi.org/10.21037/mhealth-20-119

Data Sharing Statement: Available at http://dx.doi.org/10.21037/mhealth-20-119

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/mhealth-20-119). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The usability testing protocol was approved by the West Virginia University (WVU) Institutional Review Board (NO.: FWA00005078). All participants gave informed consent prior to participation.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Cdc.gov. 2011 [cited 27 October 2017]. Available online: https://www.cdc.gov/nchs/data/databriefs/db219.pdf
2. Berrington de Gonzalez A, Hartge P, Cerhan J, et al. Body-Mass Index and Mortality among 1.46 Million White Adults. N Engl J Med 2010;363:2211-9. [Crossref] [PubMed]
3. Flegal KM, Graubard B, Williamson D, et al. Cause-Specific Excess Deaths Associated with Underweight, Overweight, and Obesity. JAMA 2007;298:2028. [Crossref] [PubMed]
4. Pantalone KM, Hobbs T, Chagin K, et al. Prevalence and recognition of obesity and its associated comorbidities: cross-sectional analysis of electronic health record data from a large US integrated health system. BMJ Open 2017;7:e017583 [Crossref] [PubMed]
5. Petrin C, Kahan S, Turner M, et al. Current attitudes and practices of obesity counselling by health care providers. Obes Res Clin Pract 2017;11:352-9. [Crossref] [PubMed]
6. Turner M, Jannah N, Kahan S, et al. Current Knowledge of Obesity Treatment Guidelines by Health Care Professionals. Obesity 2018;26:665-71. [Crossref] [PubMed]
7. Laidlaw A, McHale C, Locke H, et al. Talk weight: an observational study of communication about patient weight in primary care consultations. Prim Health Care Res Dev 2015;16:309-15. [Crossref] [PubMed]
8. Mold F, Forbes A. Patients' and professionals' experiences and perspectives of obesity in health-care settings: a synthesis of current research. Health Expect 2013;16:119-42. [Crossref] [PubMed]
9. Brown I, Thompson J, Tod A, et al. Primary care support for tackling obesity: a qualitative study of the perceptions of obese patients. Br J Gen Pract 2006;56:666-72. [PubMed]
10. Greene J, Hibbard JH. Why does patient activation matter? An examination of the relationships between patient activation and health-related outcomes. J Gen Intern Med 2012;27:520-6. [Crossref] [PubMed]
11. Hibbard JH, Mahoney ER, Stock R, et al. Do increases in patient activation result in improved self-management behaviors? Health Serv Res 2007;42:1443-63. [Crossref] [PubMed]
12. Hibbard JH, Mahoney ER, Stockard J, et al. Development and testing of a short form of the patient activation measure. Health Serv Res 2005;40:1918-30. [Crossref] [PubMed]
13. Hibbard JH, Stockard J, Mahoney ER, et al. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res 2004;39:1005-26. [Crossref] [PubMed]
14. Skolasky RL, Maggard AM, Li D, et al. Health behavior change counseling in surgery for degenerative lumbar spinal stenosis. Part II: patient activation mediates the effects of health behavior change counseling on rehabilitation engagement. Arch Phys Med Rehabil 2015;96:1208-14. [Crossref] [PubMed]
15. Barello S, Graffigna G, Vegni E, et al. 'Engage me in taking care of my heart': a grounded theory study on patient-cardiologist relationship in the hospital management of heart failure. BMJ Open 2015;5:e005582 [Crossref] [PubMed]
16. Logsdon MC, Davis D, Eckert D, et al. Feasibility of two educational methods for teaching new mothers: A pilot study. Interact J Med Res 2015;4:e20 [Crossref] [PubMed]
17. Pitts A, Faucher MA, Spencer R. Incorporating breastfeeding education into prenatal care. Breastfeed Med 2015;10:118-23. [Crossref] [PubMed]
18. Robinson JK, Gaber R, Hultgren B, et al. Skin self-examination education for early detection of melanoma: a randomized controlled trial of Internet, workbook, and in-person interventions. J Med Internet Res 2014;16:e7 [Crossref] [PubMed]
19. Sridhar A, Chen A, Forbes ER, et al. Mobile application for information on reversible contraception: a randomized controlled trial. Am J Obstet Gynecol 2015;212:774.e1-7. [Crossref] [PubMed]
20. Morgan ER, Laing K, McCarthy J, et al. Using tablet-based technology in patient education about systemic therapy options for early-stage breast cancer: a pilot study. Curr Oncol 2015;22:e364-e369. [Crossref] [PubMed]
21. Suhling H, Rademacher J, Zinowsky I, et al. Conventional vs. tablet computer-based patient education following lung transplantation--a randomized controlled trial. PLoS One 2014;9:e90828 [Crossref] [PubMed]
22. Stribling JC, Richardson JE. Placing wireless tablets in clinical settings for patient education. J Med Libr Assoc 2016;104:159-64. [Crossref] [PubMed]
23. Jensen MD, Ryan DH, Apovian CM, et al. Guideline for the management of overweight and obesity in adults: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines and The Obesity Society. J Am Coll Cardiol 2014;63:2985-3023. [Crossref] [PubMed]
24. Running a Usability Test | Usability.gov [Internet]. Usability.gov. 2014 [cited 20 October 2017 and 18 January 2019]. Available online: https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html
25. Moderating Usability Tests | Usability.gov [Internet]. Usability.gov. 2013 [cited 18 January 2019]. Available online: https://www.usability.gov/get-involved/blog/2013/04/moderating-usability-tests.html
26. Brooke J. SUS: A quick and dirty usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, et al. editors. Usability evaluation in industry. London, UK: Taylor & Francis, 1996:189-94.