Mobile App Rating Scale (User Version) for the assessment of a community health worker medical application

Introduction

Background

Noncommunicable diseases (NCDs) represent a significant global health challenge, accounting for two-thirds of all deaths worldwide, with over 75% occurring in low- and middle-income countries (LMICs) (1). In the Philippines, cardiovascular and cerebrovascular diseases are among the leading causes of mortality, reflecting a concerning trend observed across the Western Pacific region. The economic burden of NCDs is substantial, comprising 4.8% of the country’s gross domestic product in 2017. To address this pressing concern, the Department of Health (DOH) implemented the Philippine Package of Essential Non-Communicable Disease Interventions (Phil PEN), an adaptation of the World Health Organization (WHO) guidelines for managing NCDs in low-resource settings (1-3).

Despite these efforts, healthcare professionals have faced difficulties in implementing the Phil PEN program, citing the cumbersome nature of the multiple forms required for patient risk assessment and the time-consuming process, especially in high-volume settings (4). In response to these implementation challenges, Molon (5), in collaboration with the City Health Office of Muntinlupa, developed the PhilPEN Risk Stratification application (App)—a mobile App for community health workers (CHWs) based on the Phil PEN Noncommunicable Disease Risk Assessment Form. The app was created using the extreme prototyping framework (Figures 1,2)—a subset of software prototyping initially conceptualized for the development of web apps in terms of increasingly functional prototypes, but carried over into mobile health apps—and utilized open-source technologies (i.e., HTML5, CSS3, JavaScript, and Apache Cordova), while adhering to WHO guidelines and PhilHealth Circular requirements (3,5-7). However, there was an absence of a reliable methodology for evaluating the quality of health software generated through this framework.

Figure 1 Extreme prototyping framework. This figure was reused/adapted from an Open Access article (5) under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/). UI, user interface.

Figure 2 Mobile medical app development. This figure was reused/adapted from an Open Access article (5) under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/). App, application.

Rationale and knowledge gap

Facing a similar situation and acknowledging that the assessment of mobile app quality becomes increasingly important as they continue to proliferate, Stoyanov et al. (8,9) developed the (User Version) Mobile App Rating Scale (uMARS), a simple, objective, and reliable tool for classifying and assessing the quality of mobile health apps. It provides a 20-item measure that includes four objective quality subscales—engagement, functionality, aesthetics, and information quality—and one subjective quality subscale (Appendix 1).

Objective

This study aims to assess the usability of the PhilPEN Risk Stratification App using uMARS, while continuing the extreme prototyping cycle of development. The secondary objective is to evaluate whether the app could achieve an acceptable rating score (>3) on the uMARS scale. The study emphasizes the significance of quality monitoring through validated metrics in improving adoption and facilitating continuous iterative development of medical mobile apps, particularly in resource-limited healthcare environments where efficiency and effectiveness are paramount concerns.

The conceptual framework underlying this study posits that the extreme prototyping methodology, through its iterative stakeholder engagement and user-centered design approach, enhances mobile app usability as measured by validated metrics such as uMARS. Higher usability scores theoretically correlate with improved user acceptance and adoption among CHWs, who have expressed preference for technological solutions to address the cumbersome nature of the traditional paper-based NCD risk assessment forms. This adoption pathway is expected to translate into more efficient patient data collection, reduced assessment time, and ultimately improved NCD screening coverage and patient outcomes through streamlined workflow integration (5,7).

Methods

Study design and setting

This software evaluation study employed mixed research methodologies to assess the PhilPEN Risk Stratification App. Qualitative methods included CHW end-user feedback as part of continuous requirement definition, as well as linguistic validation and cognitive debriefing of the uMARS and sections of the app.

Development followed the extreme prototyping framework consisting of three phases: static prototype, dynamic prototype, and service implementation. The static prototype phase conceptualized the app’s initial design based on the Phil PEN NCD Risk Assessment Form and WHO guidelines (3,6). The dynamic prototype phase developed a working version using HTML5, CSS3, JavaScript, and Apache Cordova (10-13), incorporating linguistic validation and cognitive debriefing with potential end-users. The service implementation phase involved the finalization of the app and its deployment for wider usage and testing among CHWs and healthcare professionals (5,7). This iterative approach allowed for the gradual refinement of the app through progressively functional prototypes (7,14).

Quantitative measures were utilized to assess app quality through uMARS dimensions: engagement, functionality, aesthetics, information quality, and subjective quality. The uMARS was selected as the evaluation tool as it is the only scientifically documented and statistically tested app rating scale designed specifically for end-users. It provides a 20-item measure with four objective quality subscales and one subjective quality subscale, with scores ranging from 1 (poor) to 5 (excellent) (8,9). Studies have demonstrated its excellent internal consistency for the full scale and good levels for all subscales, with good test-retest reliability even after a 3-month delay between ratings (8,9,15,16). Inferential statistics were not planned.

This study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The study was approved by the University of the Philippines Manila Research Ethics Board (No. UPMREB 2020-270-01) and informed consent was taken from all individual participants.

The study was conducted at the aforementioned health center in Muntinlupa City, utilizing a critical case type of purposive sampling due to time and resource constraints. All 19 CHWs who fulfilled the inclusion criteria were involved in the study, having completed the basic training requirements and been recognized as the software end-users. Inclusion criteria required CHWs to have undergone training under an accredited organization, completed DOH training for NCD intervention, used the NCD Risk Assessment Form, used a smartphone with web browser, and provided written consent.

Data collection and analysis

The participating CHWs underwent a one-day orientation and training on the mobile app (Figure 3), which included reviewing the source NCD Risk Assessment Form to highlight similarities between paper and mobile versions, followed by demonstrations and simulations of app functionality. The app was installed on two representative entry-level smartphones: a Lenovo IdeaTab A3000 and a Samsung Galaxy S5 Mini running Android 4.1 or higher. The CHWs used the app in the center’s out-patient clinic to interview adult patients and stratify their NCD risk, with the intention of collecting data from 100 patients and each CHW using the app at least 6 times. The researcher confirmed proper use and interpretation of the app throughout the process, and devices were returned at the end of each clinic day for safekeeping and data privacy compliance. After using the app, the participating CHWs rated its overall quality using the uMARS.

Figure 3 Screenshots of PhilPEN Risk Stratification App. This figure was reused/adapted from an Open Access article (5) under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/). App, application.

The prototype cycle of feedback/requirement definition, system design, coding/testing (back to feedback/requirement definition) was intended to continue until an acceptable dynamic prototype with a uMARS rating of at least 3.0 was developed. If it was below 3.0, the revisions and/or feedback provided would have been reprogrammed, and the subsequent versions tested by the CHWs, collecting data from another set of patients. The CHWs would again use the uMARS to rate the overall quality of the app, and the average final mobile app rating would be recorded (Figure 4).

Figure 4 Methodology including prototype cycle of feedback/requirement definition, system design, and coding/testing. App, application; CHW, community health worker; uMARS, (User Version) Mobile App Rating Scale.

Results

The participants had an average age of 48 (range, 37–62) years with an average of 70 (range, 33–79) months in the field as a CHW. Only 18 of the 19 CHWs were able to undergo the 1-day general orientation and training. All 18 health workers were familiar with the NCD Risk Assessment Form, having populated the form and its previous iteration with patient data along with physical examination findings since 2017. However, due to coronavirus disease (COVID) restrictions, only 8 out of 18 CHWs were able to participate and use the app, collecting data from 36 patients. This sample of eight participants—average age 48.5 years, 73.5 months or approximately 6 years experience, all female Facebook users—aligns closely with the national profile where CHWs are predominantly female (92–100%), with mean ages of 50.4–50.5 years, and with the majority (72%) having 1–6 years of service (17-19).

Five more prototype mini-cycles of feedback/requirement definition, system design, coding/testing then back to feedback/requirement definition were performed during the course of the study. Code refactoring and bug fixes caught internally were excluded in these results. A brief overview of key implementations is noted in Table 1 and Appendix 2.

The app received strong ratings across objective quality dimensions of the uMARS. For Engagement, the app scored 3.53 [standard deviation (SD) =0.48, Table 2 and Table S1], with users finding it interactive and useful despite limited customization options (scoring 2.50 in A03).

Functionality received a high objective rating at 4.19 (SD =0.61, Table 2 and Table S2), with its performance (B06) metric rating highly among users indicating that the app features [e.g., automatic computation of age, body mass index (BMI), waist-to-hip (WH) ratio and risk stratification] performed accurately and as intended. Users also appreciated the gestural design (4.38 in B09), automatic calculations, and clear data entry guidance. CHWs noted that practice improved efficiency but identified some navigation challenges when correcting missed entries.

The aesthetics dimension scored 4.08 (SD =0.50, Table 2 and Table S3), with users finding the layout clear and appropriate (4.25 in C10) despite the developer’s intentionally simple design approach.

Information quality was rated highest at 4.41 (SD =0.57, Table 2 and Table S4), with the “Quantity of information” metric scoring an impressive 4.63 (D14). This section and its corresponding metric components were given priority, even if the final rating would knowingly be based on all four objective quality ratings having equal weight. CHWs valued the comprehensive risk stratification capabilities previously only performed by physicians, though some expressed they would still seek physician confirmation before acting on the app’s recommendations.

In terms of subjective quality (personal opinion), the app received a moderate score of 3.25 (SD =0.48, Table 2 and Table S5). CHWs indicated strong willingness to recommend the app to colleagues (4.13 in E17), recognizing its potential benefits for NCD management. However, when asked if they would pay for the app, responses were overwhelmingly negative (1.75 in E19), possibly influenced by the understanding that the app was being developed free of charge for their benefit. Despite this, the overall star rating was positive at 3.88 (E20), indicating general satisfaction with the app’s utility and performance.

The combined average scores for engagement, functionality, aesthetics, and information resulted in an objective quality rating of 4.05 (SD =0.37), exceeding the target score of 3.0. This rating, along with the 3.88-star subjective rating, demonstrated strong alignment around the 4.0 mark, validating the captured overall quality perception. The analysis supports Stoyanov et al.’s (8,9) assertion that uMARS can serve not only as a measurement tool but also as a development guide, helping developers anticipate challenges, make appropriate concessions, and leverage strengths to create mobile health apps that achieve acceptable quality ratings.

The overall uMARS scale also demonstrated acceptable internal consistency reliability (Cronbach’s α =0.75), indicating adequate measurement coherence across the 20 items. Subscale reliability varied considerably, with the Functionality subscale showing acceptable reliability (α =0.75), while the aesthetics (α =0.41) and personal opinion (α =0.37) subscales exhibited poor internal consistency (Table 2 and Tables S2,S3,S5). This differs from Stoyanov et al.’s excellent internal consistency for the total uMARS score (Cronbach α =0.90) and very high internal consistencies of its subscales (Cronbach α =0.71–0.80) (9).

Discussion

Key findings

The uMARS was selected as the evaluation tool for the PhilPEN Risk Stratification App because it represents the only scientifically documented and statistically tested app rating scale specifically designed for end-users (8,9,15,16). The results demonstrate that uMARS effectively gathered both objective and subjective feedback and translated it into comprehensible metrics for the developed mobile app.

The above-average scores—3.53 for engagement, 4.19 for functionality, 4.08 for aesthetics, and 4.41 for information, combining for a 4.05 objective quality rating, alongside a 3.25 subjective quality rating—testify to the effectiveness of extreme prototyping for medical mobile app development. This aligns with Stoyanov et al.’s (8) suggestion that MARS could provide mHealth app developers guidance by functioning as a checklist of criteria rather than merely an assessment tool. This study potentially represents the first in literature, outside the original authors, to advocate for this development approach.

During the initial investigation and requirement definition phases, the uMARS objective metrics proved valuable in managing developer expectations by identifying potential strengths and weaknesses. It was anticipated that the app would score lower in engagement due to limited entertainment value and in aesthetics due to the necessity for simpler graphics on entry-level smartphones. However, the app’s ability to process patient data and deliver quality information immediately after interviews presented advantages in information and functionality—areas where development efforts were consequently prioritized.

It is crucial to note that lower priority areas were not neglected entirely. The uMARS sub-questions proved instrumental during the system design and coding/testing phases. For instance, while engagement initially received less attention, its components of customization, interactivity, and target group focus were subsequently improved. Similarly, for the information category, presenting comprehensive data concisely without overwhelming users was achieved through a “comprehensive and concise” output after risk stratification, reducing extensive WHO documentation to manageable key points for healthcare workers.

Deliberate awareness of uMARS metrics served as an excellent guide throughout the development process. While the points highlighted by uMARS might seem obvious, the experience demonstrated that during prototyping cycles, critical areas can be overshadowed by competing priorities (14,20). Alongside the extreme prototyping framework, uMARS proved to be an appropriate guide for avoiding common pitfalls in medical app creation.

Limitations

The generalizability of the results is limited by the study population being significantly underpowered against the almost 300,000 CHWs nationwide (21) due to the restrictions imposed by the onset of the pandemic. As such, while the objective of an acceptable (3.0) or better rating was achieved with the current data set, it cannot be confirmed if it would have remained the same if more subjects participated or if another prototyping cycle would have been warranted. Additionally, it may have also affected the stability of the internal consistency recorded. Finally, the in-person administration of the uMARS evaluation may have introduced social-desirability bias, as participants could have been influenced to provide more favorable responses when directly interacting with the researcher, potentially inflating the overall scores and affecting the validity of the findings (22).

Conclusions

This study applied design theory to develop a mobile app supporting healthcare delivery and assessed its quality using validated metrics. Analysis of the limited data suggests that the extreme prototyping framework effectively bridges health science and information technology in medical mobile app development. The results support uMARS as a valuable guide for medical mobile app development—one of the first documented instances of this alternative use in literature. The recommended next step involves expanding the app’s usage among more CHWs to further assess its usability in clinical settings and implement improvements through additional prototype iterations. These future studies should also consider implementing anonymous survey administration through self-administered electronic or paper-based methods rather than in-person data collection, as anonymity has been demonstrated to reduce social-desirability bias and elicit more accurate responses in evaluative research contexts.

A successful integration of the PhilPEN Risk Stratification App into existing CHW workflows holds promise for addressing systemic inefficiencies in NCD risk assessment without compromising the quality of healthcare delivery. The app’s ability to perform risk stratification previously reserved for physicians, combined with its comprehensive yet concise information delivery, suggests potential for improved screening efficiency and timely patient feedback. Furthermore, enhanced data collection capabilities may provide health officers with more robust demographic and epidemiological insights to inform evidence-based program planning and resource allocation, ultimately contributing to improved population-level NCD management outcomes in resource-constrained settings.

Acknowledgments

This research was submitted in partial fulfillment of the requirements for the Degree of Master of Health Informatics at the University of the Philippines College of Medicine.

The author would like to show gratitude to Dr. Alvin Marcelo for being first mentor and informatics pioneer; Dr. Maria Teresa Tuliao and Dr. Neal Jason Argana of the city health office for their oversight; Ms. Isis Alva for all the administrative support; Dr. Iris Tan and the thesis panel for their guidance, and fellow graduate students, Philip John Sales, Isidor Cardenas, and Roy Octaviano Dahildahil for unselfishly lending their respective works for reference.

Footnote

Data Sharing Statement: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-25-22/dss

Peer Review File: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-25-22/prf

Funding: None.

Conflicts of Interest: The author has completed the ICMJE uniform disclosure form (available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-25-22/coif). The author has no conflicts of interest to declare.

Ethical Statement: The author is accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. This study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The study was approved by the University of the Philippines Manila Research Ethics Board (No. UPMREB 2020-270-01) and informed consent was taken from all individual participants.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Mendis S. Global status report on noncommunicable diseases 2014. Geneva: World Health Organization; 2014:280.
2. Kontsevaya A, Ermakova Y, Tordrup D, et al. Prevention and control of noncommunicable diseases in the Philippines. Geneva: World Health Organization; 2019:26.
3. Padilla A. Philhealth Circular - Adoption of the Philippine Package of Essential Non-Communicable Disease (NCD) Interventions (PHIL PEN) in the Implementation of Philhealth's Primary Care Benefit Package. Manila: Department of Health; 2013:6. No.0020 s. 2013. Available online: https://www.philhealth.gov.ph/circulars/2013/circ20_2013.pdf
4. Argana N. Muntinlupa NCD Response (J Molon, Interviewer). January 24, 2017. Available online: http://drive.google.com/file/d/1e56mDze71xJCw73AHQtJP2ji6eANwOdT
5. Molon JN. Extreme Prototyping for a Community Health Worker Medical Application. Healthc Inform Res 2025;31:88-95. [Crossref] [PubMed]
6. World Health Organization. Prevention of cardiovascular disease : guidelines for assessment and management of total cardiovascular risk. Geneva: World Health Organization; 2007:92.
7. Komatineni S. On Java. Reshaping IT Project Delivery through Extreme Prototyping; Nov 15, 2006 [cited Nov 11, 2019]. Available online: http://www.onjava.com/2006/11/15/reshapingitprojectdeliverythroughextremeprototyping
8. Stoyanov SR, Hides L, Kavanagh DJ, et al. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015;3:e27. [Crossref] [PubMed]
9. Stoyanov SR, Hides L, Kavanagh DJ, et al. Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth 2016;4:e72. [Crossref] [PubMed]
10. Moncur MG, Ballard P. Javascript in 24 Hours. Indiana: Pearson Education, Limited; 2012:448.
11. Myers M. A Smarter Way to Learn JavaScript. California: CreateSpace Independent Publishing Platform; 2013:254.
12. Myers M. A Smarter Way to Learn HTML & CSS: Learn it faster. Remember it longer (Volume 2). California: CreateSpace Independent Publishing Platform; 2015:258.
13. Wargo J. Apache Cordova 4 Programming. Boston: Addison-Wesley Longman, Incorporated; 2015:560.
14. Centers for Medicare and Medicaid Services. Selecting a Development Approach. Selecting a Development Approach; 2008 Mar 27 [cited Nov 11, 2019]. Available online: https://www.academia.edu/13239574/SELECTING_A_DEVELOPMENT_APPROACH
15. Domnich A, Arata L, Amicizia D, et al. Development and validation of the Italian version of the Mobile Application Rating Scale and its generalisability to apps targeting primary prevention. BMC Med Inform Decis Mak 2016;16:83. [Crossref] [PubMed]
16. 16 Azad-Khaneghah P, Neubauer N, Miguel Cruz A, et al. Mobile health app usability and quality rating scales: a systematic review. Disabil Rehabil Assist Technol 2021;16:712-21.
17. Querri AG, Ohkado A, Kawatsu L, et al. Assessment of the role of community health volunteers in delivering primary health care in Manila, the Philippines. Kokusai Hoken Iryo 2020;35:15-25. (Journal of International Health).
18. Yamaguchi Y, Palileo-Villanueva LM, Tubon LS, et al. The Experiences of Community Health Workers in Preventing Noncommunicable Diseases in an Urban Area, the Philippines: A Qualitative Study. Healthcare (Basel) 2023;11:2424. [Crossref] [PubMed]
19. Landingin QKC. Job Performance of Barangay Health Workers (BHWS): an Assessment. International Journal of Advance Research in Community Health Nursing 2024;6:22-6.
20. Rather M, Bhatnagar V. A Comparative Study of Software Development Life Cycle Models. Int J Appl Or Innov Eng Amp Manag 2015;4:23-9.
21. UNI Global Union. Philippine Care Unions Demand Passage of Magna Carta for Barangay Health Workers. UNI Global Union. 2023 [cited Jun 8, 2025]. Available online: https://uniglobalunion.org/news/magna/
22. Krumpal I. Determinants of Social Desirability Bias in Sensitive Surveys: a Literature Review. Qual Quant 2013;47:2025-47.