Evidence-based adaptation and scale-up of a mobile phone health information service

Introduction

From the earliest reviews of mobile phone interventions for health improvement in 2010 (1), the field has rapidly developed to now have sufficient evidence to support a “review of reviews” (2) and meta-analyses of pooled data (3). Many of these studies are in the area of mHealth for behavior change and reflect a varied, robust body of evidence that ranges from randomized controlled trials of adherence to antiretroviral therapy (4), contraception (5), and smoking cessation (6) to best practice review of apps for asthma (7), calorie counting (8), and pregnancy prevention (9). In light of the dramatic growth of the field and the research base, information about the successful scale-up of mHealth interventions has not kept pace.

The term “scale-up” refers to the science of taking a proven intervention and actively disseminating it throughout an entire system to achieve greater health impact. Intervention scale-up is the intentional process of working to increase the impact of evidence-based programs so that larger populations benefit from health-promoting policies and programs (10). Scaling up health programs is appropriate once there is an evidence base documenting positive benefits from the program; the mHealth field has arrived at this point in its evolution and now needs case studies to guide successful scale-up of mobile phone interventions to benefit more people and health systems. Without models of successful mHealth program scale-up, the field remains at risk of ongoing “pilotitis” that has plagued implementers for several years (11,12).

There is a large literature base recommending best practices for successful scale-up of health and development programs that is applicable to the mHealth field. Although terminology varies across frameworks and approaches (10,13,14), key components include a tested innovation with locally generated evidence; involvement of diverse stakeholders including those with responsibility for, capacity to guide, and funding to support the scale-up process; engagement with and support for those who are scaling, including end-users or beneficiaries of the scaled service; targeted packaging and dissemination that facilitates service uptake; and continuous monitoring and evaluation, learning, research, and adaptation.

This paper describes how these principles were operationalized during the scale up and adaptation of an evidence-based mHealth intervention, Mobile for Reproductive Health (m4RH). The m4RH program provides a case study of multiple aspects of scale-up: (I) vertical integration in Tanzania, (II) horizontal scale-up in Rwanda, Uganda, and Tanzania, and (III) global scale-up.

Background

m4RH is an award-winning, automated, interactive, and on-demand short message service (SMS, or text message) system that provides simple, accurate, and globally relevant information on reproductive health. One of the first mobile phone interventions for reproductive health, m4RH was developed in 2009 as part of a pilot study in Kenya and Tanzania. The pilot study aimed to gauge the feasibility of providing family planning (FP) information via text message, the reach of mobile phones as a health communication channel, and its potential impact on FP use (15). m4RH content was developed based on existing evidence-based global resources such as the World Health Organization (WHO) Global Family Planning Handbook (16) and best practices for developing health communication programs (17). Ministries of Health in both Kenya and Tanzania and their affiliated FP technical working groups (TWG) were key contributors throughout the m4RH pilot, and they reviewed and vetted the content to ensure that it was contextually relevant.

Results from the pilot study demonstrated that m4RH engages diverse audiences including young people, men, couples, and women of reproductive age (15,18). m4RH users reported that the mobile phone platform is a highly acceptable format for receiving FP information and they appreciated the convenience and privacy of this method of communication. These results suggest that mHealth provides one avenue through which to engage populations that have been traditionally difficult to reach with FP information, including youth and men. Some m4RH users reported increased knowledge, improved partner communication, and increased contraceptive method uptake after using the system. In addition, a randomized controlled trial conducted by Abt Associates in Kenya in 2013 demonstrated a 13% improvement in contraceptive knowledge after three months among m4RH users compared to a control group (19).

The mHealth adaption model (20)

Since the pilot in 2009, m4RH has expanded its content, geographical reach, and target populations, and has been adopted and adapted by organizations around the world. To guide the process for developing content and adapting m4RH for new populations, settings, and technical areas, the team developed and followed a 10-step model grounded in research utilization and communications science and literature (13,14,17,21) (Figure 1). The model centers on the interplay between stakeholders and end-users of the mHealth program through early and ongoing stakeholder engagement and continuous data collection with mHealth program users throughout the life of the program.

Figure 1 mHealth adaptation model. The mHealth Adaptation Model is a 10-step model used to guide development and adaption of the m4RH program for new populations and settings. The evidence-based model highlights continual interaction between stakeholders and end-users of the mHealth program.

Step one: establish mobile messaging technical working group

Research shows that stakeholders who are involved in the adaptation process from the beginning are more likely to support its scale up and that stakeholder engagement ensures the intervention is relevant, appropriate, feasible, and sustainable (14,22). To ensure that relevant stakeholders are engaged in the adaptation process, the m4RH team first establishes a TWG. Members of the TWG assist in identifying key health issues and content, provide technical review and input, and support the dissemination, roll-out, scale-up, and sustainability of m4RH.

Step two: identify priority health issues and appropriate delivery format

Conducting an assessment of the target audience’s information needs and community culture is a critical step in program adaptation (17,23). This process results in small to moderate differences in program design across each country because content is selected through a careful review of research and other documentation on the information needs of the target population in each setting, a review of existing global and local health curricula, and extensive input from members of the TWG. Health information provided by existing behavior change communication programs along with funder priorities also are given careful consideration in guiding the selection of content.

Step three: develop new content as needed

A key component of program adaptation is maintaining sufficient fidelity. When adapting an intervention, it is imperative to ensure the key program elements, guiding behavioral theories, and internal logic are maintained (21,24). m4RH message development is guided by best practices in health communication and behavior change theory (17) and these same systematic processes are followed in developing new content. This ensures that new content maintains internal logic and key program elements while being responsive to new users and communities.

Step four: review content with stakeholders

Once all new content is developed, it is presented to the TWG for review. TWG members also approve program content again just before program launch and promotion in Step 8. Continuous stakeholder engagement ensures the program meets all information needs and is culturally appropriate and acceptable to the community.

Step five: test content with target audience

To ensure that program content is easily understood and contextually relevant, it is systematically tested via Focus Group Discussions (FGDs) with the target population, and key influencers (such as parents or partners) when appropriate (25). FGDs should include questions about the target groups’ mobile phone use habits, preferred health topics, feedback on the proposed program, and channels for promotion. FGDs also should provide the opportunity to review and comment on all proposed program content. Research shows that as few as 3–4 FGDs are typically necessary to reach saturation of study themes (26); and our experience suggests that each SMS message should be tested a minimum of two times across the total number of FGDs held.

Step six: program technology platform in appropriate delivery format

In each country an appropriate technological partner is identified early to code and deploy the adapted m4RH system. Following best practices in digital development, m4RH seeks to use open source technology, to build upon existing systems if possible, and to encourage collaboration among digital health and technology partners (27).

Step seven: test user interface and near-final content with target audience

Usability testing is a process in which potential users of the technology are asked to navigate through a live system and are queried about the design, navigation, and ease of use of the technology (28). After the m4RH technology platform is programmed, the system is tested with end-users in one-on-one interviews before it is launched to ensure that the system is functional, comprehensible, and easy to use.

Step eight: finalize content and platform

All user feedback gathered from content and usability testing is incorporated into the program before launch. Stakeholders review and approve the final program.

Step nine: launch and promote

The integral relationships, established at the onset of m4RH adaptation, are essential to ensure the adapted program is both promoted and available to target groups. m4RH is promoted via mass and social media, in health and FP clinics, by local health workers and during community events, and through interpersonal communication.

Step ten: monitoring and evaluation

By nature, mobile phone programs allow for continuous real time data collection. The m4RH platform is able to capture all system queries, or “hits,” through electronic and automatic logging. These data can be aggregated on a monthly, quarterly, or annual basis as needed—allowing for continuous monitoring and evaluation. Additionally, SMS data collection with users is recommended as a rapid and cost-effective method of learning more about users and the mHealth platform functionality (15,18). Other data collection methods such as phone and clinic intercept interviews provide additional sources of user and platform data.

Vertical scale in one country

Close working relationships established with key partners in Tanzania helped ensure m4RH’s success and supported its vertical integration—that is, fostering institutionalization within national programs, policies and systems and ownership among government stakeholders (10)—as a key pillar of the country’s FP program over the past 6 years. The FP TWG, convened by the MOH and including international and national FP partners, provided key program inputs. During the pilot, the FP TWG articulated local FP and reproductive health needs and priorities, provided technical expertise and contextual experience in reproductive health, offered suggestions for effective service promotion, and linked m4RH to service delivery organizations.

Stakeholders received regular data updates showing m4RH program utilization, and these data were used for decision-making around FP programming. The FP TWG frequently discussed and interpreted m4RH system data, noting for example, that content on natural FP methods was highly requested and prompting a review of the current availability of these methods within Tanzania. Based on promising results from the pilot study (15), partners endorsed expanding m4RH content to include deeper messaging on side effects and other issues related to uptake and continuation of FP methods. As was done in the pilot, stakeholders in the FP TWG reviewed global guidance documents as well as locally available data regarding barriers to FP use and commonly held misconceptions in order to develop new content, and partners also collaborated to develop the new content.

m4RH was vertically integrated into government programs and systems when the MOH endorsed promotion of the m4RH service through a national-level, government-approved FP mass media campaign. This resulted in an immediate sixteen-fold increase in queries to the m4RH system and subsequently elevated m4RH utilization levels. The MOH and the FP TWG continue to receive regular updates in the form of m4RH system data and results of user surveys. For example, a 2012 SMS survey of 25,000 m4RH users over a 6-month period illustrates the truly national reach of m4RH, with geospatial analysis of these data documenting users in 98% of the country’s districts (Figure 2). By October 2016, m4RH had reached over 500,000 users in Tanzania who have accessed the system more than 4.5 million times.

Figure 2 m4RH Tanzania Users by District, 2012. A geospatial analysis of data collected via SMS from 24,742 m4RH users in Tanzania during a 6-month period in 2012. The color of the circles indicates the number of users in each district, and the boundaries indicate the district borders. The map shows that m4RH reached individuals in 127 of 129 Tanzanian districts during the six-month data collection period.

Horizontal scale-up across three countries

The horizontal scale-up of m4RH, which encompasses geographic expansion as well as diversification (14), has occurred in Rwanda, Uganda and Tanzania. Based on pilot data showing young people were frequent users of the m4RH system, these countries chose to adapt the m4RH system to reach key adolescent and youth audiences ages 10–24. Following the 10-step model shown above, the m4RH platform in these settings has broadened to include additional content on puberty, sex, pregnancy, gender based violence, HIV, and STIs. The new content also includes role model stories that model positive sexual and reproductive health behaviors and service utilization among young people. Common elements that facilitated scale-up in each country include: key involvement from Ministries of Health, Youth, Gender, and Education; formation of the m4RH TWG; incorporation of country-specific health resources; engagement with parents, caregivers, and community members; and co-creation of the technical content and program platform with target audiences.

Focus on Rwanda

In Rwanda, the MOH requested that the m4RH team adapt content for young people and collaborate with a local technology partner to develop the platform, with the understanding that the MOH and local health partners would then deploy the program. An m4RH TWG was formed to support program adaptation, platform development, handoff, and national scale-up from the outset. The TWG supported (I) identifying the key sexual and reproductive health and gender issues for young people in Rwanda; (II) identifying the most appropriate message format for youth; (III) reviewing, providing technical input, and endorsing final content; and (IV) translating text messages into the local language.

The involvement of young people, parents, caregivers, and other influential adults such as religious and community leaders, was instrumental to the adaptation process. FGDs with parents and other adults in each country helped to ensure program acceptability among key gatekeepers in young people’s lives. FGDs with youth allowed for the m4RH program content to be tailored in a way that was relevant to youth, that was perceived as trustworthy and accurate, and that supported culturally appropriate and healthy decision making (25).

In Rwanda, 10 FGDs were conducted with 15–24-year-old males and females stratified into same-gender groups of 15–17 and 18–24 years old. Young people who were comfortable text messaging were recruited from youth centers. Highly trained research assistants who had experience working with young people facilitated FGDs. Focus groups were recorded, transcribed, and analyzed thematically, and data was discussed and interpreted with the full research team to yield final results. FGD participants reported that m4RH would be a good program for youth in Rwanda, and that the information provided will empower them and support their decision-making (Table 1). Four FGDs also were conducted with Rwandan adults recruited from market places and community centers. Adult participants stated that they would recommend the program to youth and that they would be happy to see their children using m4RH. They reported that they would encourage their children to explore all of the content and some even felt it would help facilitate communication about difficult subjects.

Table 1 Quotes illustrating parent and young peoples’ perceptions of the m4RH program in Rwanda
Full table

Finally, usability testing helped to ensure that the program content was delivered in a way that was accessible to the younger target population (28). Twenty one-on-one interviews were conducted with 15–24 years old. IDI participants were asked to navigate through a live technological system, observed for how they use the system, and then asked a series of questions about the design, navigation, and use of the system. Usability testing was especially important in the Rwanda m4RH adaptation because a new local technology partner was engaged for platform development and new functionalities, including USSD, were being considered. The flexibility and simplicity of the m4RH system–which were intentional design features—allow for its easy adoption by multiple technology partners, and therefore testing the user interface and navigation along with ensuring platform functionality when deployed by a new partner was essential.

Global scale-up

Beyond vertical and horizontal scale-up, the m4RH program has scaled within the global public health community by providing tools and content to enable others to build their own adapted and evidence-based mHealth programs. Strategic partnerships, effective data sharing, and application of marketing practices have been used to champion broad adoption of m4RH along with best practices in development and implementation of mobile phone interventions for health globally.

Since program inception, the m4RH team has attended global and local FP conferences and mHealth meetings to share the program; held online and in-person consultations with colleagues based in academic, quasi- and governmental settings, and other non-governmental organizations (NGOs); and served as technical advisors to other programs and countries interested in implementing m4RH or related mHealth programs. These conversations have informed partner development of multiple mobile phone programs for reproductive health including the youth-focused global ARMADILLO program (29), mCenas in Mozambique (30), and ChatSalud in Nicaragua (31). They have also facilitated the forthcoming integration of m4RH into programs beyond the FP sector, including a maternal health program in Afghanistan and an education program in Malawi. In addition, a series of peer-reviewed journal articles have been published on m4RH that detail program content, reach, and impact (15,18,19), provide methods for adapting and piloting m4RH for young people (29), and model financial costs for sustaining m4RH (32).

The availability of peer-reviewed data and program results directly addresses challenges to implementing evidence-based programs and policies, notably a significant time-lag in publishing results (33). Furthermore, because published manuscripts often lack sufficient information about the intervention to facilitate replication (34,35), the m4RH team has explicitly focused on packaging m4RH to make it available for partners to adapt and apply. The m4RH team has produced several toolkits and guides that provide concise, easy-to-read and understand content showcasing the mHealth platform, content, evidence, and the adaptation process (20). Data from m4RH users documenting willingness to pay for the program content, along with details about program costs, also has been featured in project materials (20,32). With this information available, program implementers can make an informed decision as to whether m4RH is appropriate to their context and needs. Finally, we have developed a simple, non-monetary m4RH licensing model that empowers others to build upon m4RH for their own mHealth programs. m4RH license agreements have been signed with NGOs operating around the world in Bangladesh, Kenya, Mozambique, Myanmar, Nicaragua, Nigeria, and Tanzania, among others, providing these partners with full access to the m4RH content and internal logic and the right to create derivative works.

Conclusions

The m4RH case study highlights the essential elements of mHealth program development and adaptation that contributed to its successful vertical, horizontal, and global scale-up. The adaption and scale-up of m4RH demonstrates how program monitoring and research, targeted dissemination, partnerships, and strategic design are essential for developing and leveraging platforms to carry forward and expand mHealth services.

When developing new intervention approaches, especially in emerging fields like mHealth where questions about reach, usability, and efficacy are paramount, pilot studies are essential (11,36,37). However, single data collection events are not enough to inform decisions about adaptation and scale; data collection must be continuous and results must be readily available to generate the necessary stakeholder support for scale up (38). The m4RH service was originally developed as a pilot program to evaluate the feasibility and potential impact of an mHealth FP information service (15). Data obtained during the m4RH pilot answered several fundamental stakeholder questions and informed decisions to scale the program: data showed that important but hard-to-engage target audiences were reached by m4RH messaging, that end-users liked the mHealth program and wanted deeper content, and that partners were enthusiastically promoting the program as a resource not only for their beneficiaries but also for their health workers. Program monitoring, evaluation, and iterative learning and development should be prioritized in all health and mobile phone interventions.

Providing access to program materials and targeting key audiences for data dissemination were critical for scale-up. Publishing in peer-review journals, attending FP and global health conferences, sitting at the table during TWG meetings, responsively answering calls for more information from any and all actors, and fee-free licensing of m4RH were key advocacy tools. Emphasizing stakeholder engagement throughout the adaptation and scale-up process and ensuring relevance and reliability by providing ongoing opportunities for input from stakeholders, target audiences, and community members was critical. This open access approach and emphasis on partner relationships fostered local program adoption and ownership and supported sustainability through increased and diversified program funding—key building blocks of scale-up (10).

A key tenet of scale-up is to “begin with the end in mind” (14) and the m4RH team comprehensively embraced this perspective. Initial m4RH content was based on global guidelines for FP to ease the adaptation of messaging for new populations, languages, and countries. Similarly, the 10-step m4RH adaptation model was developed to be equally relevant across new health domains and content, populations, and stakeholder groups. In addition, the original technology partnership stated that the program code could be shared with other technology groups as needed, which provided flexibility for working with local and global technology groups.

More research is required to answer questions about costs and benefits of mobile phone interventions for health improvement to better understand whether and how to scale (39,40). While we have explored financial sustainability of m4RH by developing and evaluating a series of cost-recovery models (32), there are few cost benefit analyses of mHealth programs. Along with development of case studies on mHealth scale-up, mHealth costing studies must be prioritized.

This manuscript is one of the few papers to document scale-up of an mHealth intervention across multiple countries and audiences. The m4RH program was early in its use of mobile phones for population health improvement, but systematic in its application of best practices for scaling health and development programs. The evidence-based mHealth Adaption Model for adapting and scaling mHealth programs can inform the successful scale-up and sustainability of mHealth programs that are relevant across health topics and settings. Applying this model could empower health professionals and teams to build mHealth programs that are poised for success and may help to accelerate health impact of mobile phone interventions.

Acknowledgements

Christine Lasway, Heather Vahdat, FHI 360 country teams, and Text to Change were vital to the success of the m4RH program. We thank all of the wonderful partners who have worked with the m4RH team in the past, present, and future.

Funding: m4RH initially was funded by USAID (Cooperative Agreement GPO-A-00-08-00001-00), with additional funding through USAID Missions in Tanzania, Rwanda, and Uganda to FHI 360 through the Progress and Advancing Partners and Communities projects. m4RH program dissemination was supported with a grant to FHI 360 (Award #UNF-13-544) from the Innovation Working Group Catalytic mHealth Grant Program funded by the Norwegian Agency for Development Cooperation (Norad) as part of the commitment to the UN Secretary General’s Every Woman Every Child Initiative.

Footnote

Conflicts of Interest: K L’Engle was with FHI 360 at the time this work was carried out. FHI 360, 359 Blackwell Street, Durham, NC 27701, USA. The other authors have no conflicts of interest to declare.

Disclaimer: The views in this work are solely the responsibility of the authors and do not necessarily represent the views of the funders.

References

1. Cole-Lewis H, Kershaw T. Text messaging as a tool for behavior change in disease prevention and management. Epidemiol Rev 2010;32:56-69. [Crossref] [PubMed]
2. Hall AK, Cole-Lewis H, Bernhardt JM. Mobile text messaging for health: a systematic review of reviews. Annu Rev Public Health 2015;36:393-415. [Crossref] [PubMed]
3. Orr JA, King RJ. Mobile phone SMS messages can enhance healthy behaviour: a meta-analysis of randomised controlled trials. Health Psychol Rev 2015;9:397-416. [Crossref] [PubMed]
4. Lester RT, Ritvo P, Mills EJ, et al. Effects of a mobile phone short message service on antiretroviral treatment adherence in Kenya (WelTel Kenya1): a randomised trial. Lancet 2010;376:1838-45. [Crossref] [PubMed]
5. Castaño PM, Bynum JY, Andrés R, et al. Effect of daily text messages on oral contraceptive continuation: a randomized controlled trial. Obstet Gynecol 2012;119:14-20. [Crossref] [PubMed]
6. Free C, Knight R, Robertson S, et al. Smoking cessation support delivered via mobile phone text messaging (txt2stop): a single-blind, randomised trial. Lancet 2011;378:49-55. [Crossref] [PubMed]
7. Househ M, Hossain N, Jamal A, et al. A cross-sectional content analysis of Android applications for asthma. Health Informatics J 2016. [Epub ahead of print]. [Crossref] [PubMed]
8. Davis SF, Ellsworth MA, Payne HE, et al. Health Behavior Theory in Popular Calorie Counting Apps: A Content Analysis. JMIR Mhealth Uhealth 2016;4:e19. [Crossref] [PubMed]
9. Chen E, Mangone ER. A Systematic Review of Apps using Mobile Criteria for Adolescent Pregnancy Prevention (mCAPP). JMIR Mhealth Uhealth 2016;4:e122. [Crossref] [PubMed]
10. ExpandNet, World Health Organization. Nine steps for developing a scale-up strategy. Geneva, Switzerland: World Health Organization; 2010.
11. Labrique A, Vasudevan L, Chang LW, et al. H_pe for mHealth: more "y" or "o" on the horizon? Int J Med Inform 2013;82:467-9. [Crossref] [PubMed]
12. Tomlinson M, Rotheram-Borus MJ, Swartz L, et al. Scaling up mHealth: where is the evidence? PLoS Med 2013;10:e1001382. [Crossref] [PubMed]
13. ExpandNet, Management Sciences International, World Health Organization. Twenty Questions for developing a scaling-up case study. Geneva, Switzerland: World Health Organization; 2007.
14. World Health Organization, ExpandNet. Beginning with the end in mind: Planning pilot projects and other programmatic research for successful scaling up. Geneva, Switzerland: World Health Organization; 2011.
15. L'Engle KL, Vahdat HL, Ndakidemi E, et al. Evaluating feasibility, reach and potential impact of a text message family planning information service in Tanzania. Contraception 2013;87:251-6. [Crossref] [PubMed]
16. World Health Organization Department of Reproductive Health and Research, Johns Hopkins Bloomberg School of Public Health/Center for Communication Programs. Family planning: A global handbook for providers. Geneva, Switzerland: World Health Organization; 2011.
17. U.S. Department of Health and Human Services. Making health communication programs work. Washington, D.C.: US Department of Health and Human Services; 2003.
18. Vahdat HL, L'Engle KL, Plourde KF, et al. There are some questions you may not ask in a clinic: providing contraception information to young people in Kenya using SMS. Int J Gynaecol Obstet 2013;123 Suppl 1:e2-6. [Crossref] [PubMed]
19. Johnson D, Juras R, Riley P, et al. A randomized controlled trial of the impact of a family planning mHealth service on knowledge and use of contraception. Contraception 2017;95:90-97. [Crossref] [PubMed]
20. FHI 360. m4RH Toolkit. Washington, DC: FHI 360; 2016.
21. Kilbourne AM, Neumann MS, Pincus HA, et al. Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement Sci 2007;2:42. [Crossref] [PubMed]
22. McKleroy VS, Galbraith JS, Cummings B, et al. Adapting evidence-based behavioral interventions for new settings and target populations. AIDS Educ Prev 2006;18:59-73. [Crossref] [PubMed]
23. Bell SG, Newcomer SF, Bachrach C, et al. Challenges in replicating interventions. J Adolesc Health 2007;40:514-20. [Crossref] [PubMed]
24. Wingood GM, DiClemente RJ. The ADAPT-ITT model: a novel method of adapting evidence-based HIV Interventions. J Acquir Immune Defic Syndr 2008;47 Suppl 1:S40-6. [Crossref] [PubMed]
25. Yardley L, Morrison L, Bradbury K, et al. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res 2015;17:e30. [Crossref] [PubMed]
26. Guest G, Bunce A, Johnson A. How many interviews are enough: an experiment with data saturation and variability. Field Methods 2006;18:59-82. [Crossref]
27. Principles for Digital Development. Available online: http://digitalprinciples.org
28. Zhang D, Adipat B. Challenges, methodologies, and issues in the usability testing of mobile applications. Int J Hum Comput Interact 2005;18:293-308. [Crossref]
29. Gonsalves L, L'Engle KL, Tamrat T, et al. Adolescent/Youth Reproductive Mobile Access and Delivery Initiative for Love and Life Outcomes (ARMADILLO) Study: formative protocol for mHealth platform development and piloting. Reprod Health 2015;12:67. [Crossref] [PubMed]
30. Pathfinder International. Assessing the Effects of mCenas! SMS Education on Knowledge, Attitudes, and Self-Efficacy Related to Contraception in Mozambique. Boston, MA: Pathfinder International; 2015.
31. MAMA Community Spotlight: ChatSalud Nicaragua. Available online: http://www.mhealthknowledge.org/sites/default/files/Spotlight-Chatsalud.pdf
32. Mangone ER, Agarwal S, L'Engle K, et al. Sustainable Cost Models for mHealth at Scale: Modeling Program Data from m4RH Tanzania. PLoS One 2016;11:e0148011. [Crossref] [PubMed]
33. Balas EA. From appropriate care to evidence-based medicine. Pediatr Ann 1998;27:581-4. [Crossref] [PubMed]
34. Glasgow RE, Emmons KM. How can we increase translation of research into practice? Types of evidence needed. Annu Rev Public Health 2007;28:413-33. [Crossref] [PubMed]
35. Rimer BK, Glanz K, Rasband G. Searching for evidence about health education and health behavior interventions. Health Educ Behav 2001;28:231-48. [Crossref] [PubMed]
36. Agarwal S, LeFevre AE, Lee J, et al. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 2016;352:i1174. [Crossref] [PubMed]
37. World Health Organization. mHealth: New Horizons for Health Through Mobile Technologies. Geneva, Switzerland: World Health Organization; 2011.
38. Adamou B, Curran J, Wilson L, et al. Guide for Monitoring Scale-up of Health Practices and Interventions. Available online: https://www.measureevaluation.org/resources/publications/ms13-64
39. Schweitzer J, Synowiec C. The economics of eHealth and mHealth. J Health Commun 2012;17 Suppl 1:73-81. [Crossref] [PubMed]
40. Fritz F, Kebede M, Tilahun B. The need for cost-benefit analyses of eHealth in low and middle-income countries. Stud Health Technol Inform 2015;216:981. [PubMed]