Mobile training and support (MOTS) service—using technology to increase Ebola preparedness of remotely-located community health workers (CHWs) in Sierra Leone

Introduction

West Africa was struck in 2014–2016 by the largest outbreak of Ebola to date. Over 11,000 cases of Ebola were registered across Guinea, Liberia and Sierra Leone, with about 4,000 deaths in Sierra Leone (1). Many international bodies sought ways to fight the epidemic and as part of the European Community’s efforts to address the advancement of Ebola, a call for projects was opened by the Innovative Medicines Initiative (IMI). A public-private partnership called EBODAC (2) was funded out of this IMI-initiative. This led to the formation of a consortium composed of London School of Hygiene and Tropical Medicine, Janssen Pharmaceutica N.V. (part of the Janssen Pharmaceutical Companies of Johnson & Johnson), World Vision and Grameen Foundation. The consortium’s goal was to develop strategies and tools to promote the acceptance and uptake of new candidate Ebola vaccines. The consortium has supported candidate Ebola vaccine trials and beyond this, they also have a remit to build local knowledge and capacity in preparation for the potential future use of a licensed Ebola vaccine.

As part of this latter mandate, the EBODAC consortium developed a Mobile Training and Support (MOTS) service that was piloted with Community Health Workers (CHWs) in Bo district, Southern Sierra Leone in 2018. CHWs are recognized to have played a critical role in frontline health delivery during the Ebola outbreak in Sierra Leone (3). The Sierra Leonean Ministry of Health & Sanitation (MoHS) has also invested into strengthening the knowledge and performance of the country’s CHW population with the introduction of a new national community health worker policy and associated training curriculum in 2016 (4). The EBODAC consortium has since forged a partnership with the MoHS to develop, test and roll out an innovative way to provide refresher trainings on vaccinations and disease surveillance to a large group of remotely located CHWs, with a special focus on Ebola. This partnership led to the selection of Interactive Voice Response (IVR) technology as a means to deliver audio-based refresher trainings on the topics of vaccines and outbreak response including Ebola disease surveillance procedures. Training modules and associated quiz elements were made available to selected CHWs via IVR. This approach was considered as a possible additional tool to supplement classroom training approaches for educating CHWs but the utility of course still needed to be assessed with the piloting of the platform.

The focus of this manuscript is therefore to address the question of whether the chosen IVR technology can play a role in health education for remotely-located frontline CHWs. The primary objective is to determine whether this technological approach to training is well accepted by the CHWs and other key stakeholders while the secondary objective is to address whether CHWs following trainings with MOTS acquire additional or improve Ebola-related knowledge.

Methods

Prior to undertaking any detailed design work for the training platform, a preparative assessment by World Vision to examine the functional capabilities of CHWs on the ground was an important input into the software requirements. This yielded learnings on the level of education of the target population, required languages to ensure comprehension of the key training messages, ownership of a functioning mobile phone and the level of mobile connectivity in the locations (5). These elements were included in the final MOTS platform design to ensure fit for purpose delivery.

Collaboration was an underlying theme in the successful development of the MOTS training platform. Collaboration between consortium partners was key but even more instrumental was the collaboration with the local stakeholders: Sierra Leonean MoHS specifically the National CHW and e-Health coordination hubs

Upon review of the consortium’s concept note, the MoHS suggested the formation of a working group together with members from the e-health and CHW-hub. This working group ratified the concept of how MOTS should and could work considering the Sierra Leone setting as per the illustration below (Figure 1).

Figure 1 Schematic showing how MOTS works. MOTS, mobile training and support.

General requirements

The technology partner Grameen Foundation undertook the development of the platform based upon requirements gathered by the broader consortium and local Sierra Leone stakeholders. Consideration was given in the requirements to the fact that the training modules when delivered in audio format needed to be concise to ensure greater probability of completion. As a result, each training module was divided into discrete units, each with a limited duration of at most 5 minutes.

Further, what was considered critical to the development was the key requirement that the platform be sustainable, hence fitting into the National Digital Health Strategy. Alignment with the strategy would determine the need for MOTS to be able to interact with other existing government systems like the Rapidpro platform currently used by the MoHS. Ensuring accessibility to MOTS in a sustainable way in the long term was foreseen by offering the software code that was developed into an open source community. With this goal in mind, the MOTS software code, which was developed using Java and Mobile Technology for Community Health software (MOTECH), was made publicly available on GitHub (https://github.com/motech-implementations/mots).

Technical design

The platform was ultimately composed of two main parts: The Training administration component and the Reporting component. The Training administration interface is used to register CHWs on the platform and to allocate them training modules while the reporting dashboard provides a decentralized reporting functionality to in-charges (supervisors of the CHWs) and other managerial levels to enable tracking progress of the training program and monitoring the success of training at an individual CHW level (Figure 2).

Figure 2 Schematic showing the technical structure of MOTS. MOTS, mobile training and support.

Multiple reports were developed for the reporting dashboard such as CHW training module progress report, CHW module unit progress report and CHW quiz report amongst others. Reports can also be downloaded into PDF or Excel format (Figure 3).

Figure 3 Example of a progress report at unit level.

Training content

Given the intent to ensure better preparedness of CHWs in Sierra Leone should a new outbreak of Ebola occur, it was decided to focus one training module around basic vaccine knowledge (module I) so that CHWs are reminded of the importance of vaccinations, that they gain understanding of what vaccines are, why they are required, when they can be administered and how communities can access standard vaccinations that are foreseen nationally. The importance of underlining the value of vaccines preempts any possible deployment of a licensed vaccine for Ebola and prepares CHWs for this eventuality. A second training module was centered around disease surveillance & outbreak response (module II) and reminds CHWs on the steps they need to take when detecting communicable diseases like Ebola according to national guidelines.

The basis of all training content was the MoHS’ curriculum for CHWs. This was complemented by expertize of The Network for Education & Support in Immunisation (NESI: http://www.nesi.be/) with proven experience in immunization training curriculum development and delivery in low- and middle-income countries. NESI ensured augmentation of the main training curriculum with other independent sources, the correctness and synchronization with the in-country procedures and above all, independent validation of the training content. This validation step together with health promotion department of the MoHS ensured that the training content was fully endorsed by the MoHS.

Each training module was ultimately divided into discrete units to ensure that CHWs could take the training in bite-size amounts in a flexible way that fits easily into their schedule. The CHW hub and Health promotion department provided advice as to the maximum duration of any given unit such that voice files could be no longer than five minutes and a full module of voice files designed to refresh key messages from the face-to-face national curriculum training could not exceed 30 minutes and take no more than four weeks to complete. All trainings were offered in the preferred local languages of the CHWs (Krio, Mende, or English) and local names were used. All audio files with the key training messages were recorded by local voice artists. All voice files were quality-controlled and back-translated to ensure correct translation and clarity of the training content. Field testing of translated and recorded content were also done to inform the final message development.

Each module contained a pre-training quiz to assess the level of the CHWs knowledge on the topic of the module. Once all the units in a module were completed, a summary training unit encompassing a post training quiz was foreseen. Quizzes were set up with situational vignettes depicting plausible scenarios the CHWs could confront in the field. CHWs were then asked a multiple-choice question.

An example of such a vignette is as follows:

• Sallay, the CHW, visited the household of Isatu after learning that her five-year old daughter, Fanta, has a fever and a rash since a few days ago. Sallay asked for additional information and to see Fanta to look at her condition.

Press 1 if Sallay should report that Fanta has suspected measles.

Press 2 if Sallay should report that Fanta has suspected Ebola.

Press 3 if Sallay should report that Fanta has yellow fever.

A limited visual aid based on the CHW national policy document was given to the CHWs in accompaniment to the training modules.

The EBODAC consortium developed a metrics & evaluation framework to help assess the effectiveness of the platform in bringing incremental knowledge to the CHWs on the topics trained. The key elements in this framework addressed the degree of knowledge acquisition realized by the CHWs upon following the refresher training. Further, qualitative perceptions of the key stakeholders (CHWs, their supervisors, members of the Ministry of Health & Sanitation, Household members receiving information from the CHWs on the topics of the training modules) using the MOTS platform were captured by way of interviews and focus group discussions.

A location to pilot MOTS was determined in alignment with the Sierra Leonean MoHS. Their advice led the consortium to consider the district of Bo as the pilot site to test MOTS with local CHWs (Figure 4).

Figure 4 Map illustrating the location of the MOTS pilot in Bo district in Sierra Leone. MOTS, mobile training and support.

Bo district is the second most populous district in Sierra Leone with a population of over five hundred and seventy-five thousand inhabitants (5). A listing of CHWs to be considered for eligibility for the pilot was provided by the MoHS to the EBODAC consortium.

Simple eligibility criteria were applied—these included: The CHW’s willing to fully participate in the pilot, must be reporting to the Peripheral Health Unit (PHU) within Bo district, have a functional mobile phone and should be based in the geographical location of the PHU assigned.

In line with the Declaration of Helsinki 2013, approval for the pilot protocol was obtained from the Government of Sierra Leone Ethics and Scientific Review Committee, Directorate of Policy, Planning & Information prior to initiation of any efforts in Bo District. Once this ethics approval was obtained, a CHW functionality assessment was carried out to inform stakeholder engagement approach. Sensitization work in the local community was undertaken to ensure the key stakeholders were fully aware of the project and engaged to support the effort. Sensitization took the form of household visits and interviews with in-charge supervisors of the CHWs and the CHWs themselves. All CHWs provided informed consent for pilot participation. Households theoretically served by CHWs were interviewed as well as they were considered the independent evaluators of the CHWs’ ability to provide information on vaccines and disease surveillance.

Orientation was finally undertaken to explain how the MOTS platform worked. Hands-on practice was given to CHWs so that they knew how to engage with the platform using their basic feature phones while training for in-charge supervisors and managerial representatives from the MoHS was focused towards how to use the reports from the reporting dashboard to track progression of the training.

Results

Demographics of pilot participants

The functional assessment of 264 CHWs in the catchment area of Bo district covered the five chiefdoms of Bagbo, Bumpe-Ngao, Jiama Bongor, Lugbu and Tikonko. Not all 264 CHWs were able to participate in the pilot. Despite all 264 CHWs being based in the geographical area for the pilot duration and 99% being willing to participate in the MOTS pilot, only 82% of the CHWs had a functional phone with a clear screen, and 79% had a phone with functional keys. Further, 47% of the CHWs were deemed active in their community on a monthly basis and therefore included in the study (6). Application of all these eligibility criteria resulted in a final target group of 125 consented CHWs. Two of these CHWs later became ill and did not fully complete the MOTS pilot assessment (Figure 5).

Figure 5 Profile of CHWs selected to participate in MOTS pilot in Bo district. CHWs, community health workers; MOTS, mobile training and support.

The majority of the 125 CHWs were below the age of 50 years, with 21 percent falling below the age of 35. More male CHWs (n=101) than female CHWs (n=24) were included in the pilot. This cluster of CHWs reported to 25 Peripheral Health Units (PHUs) in the Bo district of which, 10 were Maternal and Child Health Posts (MCHPs), eight CHP Community Health Posts (CHPs) and seven Community Health Centers (CHC’s). Eighty percent of CHWs engaged in crop production within the geographical location. All CHWs had formal education with a majority (53%) having up to Senior Secondary education, and 28% having up to junior secondary education.

Knowledge change

Based upon the assessments made both pre- and post-training across the total CHW population, an increase in knowledge acquisition was observed for each module trained. For module I (on vaccines), at the pre-training test approximately 10 percent of the CHWs answered at least 80 percent of the questions correctly, whereas at the post-training test, almost 40 percent did (Figure 6), representing a 30 percent point change. For module II (on disease surveillance and outbreak response), 21 percent of CHWs answered more than 80 percent of the questions correctly, whereas as the post-training test 24 percent answered correctly, representing only a three percentage point change (Figure 6). Interestingly, the percentage of CHWs answering between 50 to 79 percent of the questions correctly showed some regression post-training. The extent of knowledge change was higher with the vaccine training compared to the disease surveillance and outbreak response (Ebola) module.

Figure 6 Illustration of knowledge achievement score for training module I vs. module II.

When a within subject analysis was carried out i.e., looking at an individual CHW’s performance at pre- versus post-training, and removing any CHWs with missing data from the pre- or post-training assessment, for module I, 67% (20/30) of the CHWs had an improved post-training total score i.e., they improved with at least one point in their total score post-training relative to their baseline assessment while the range for potential improvement on both modules was between 0 to 20. For module II, 64% (44/69) of the CHWs had an improved end-line total score.

In training module I, the percentage of CHWs responding correctly to questions ranged between 47–76% while in module II this percentage was between 58–72%. The percentage of CHWs responding incorrectly ranged between 11–29% for module I and between 18–37% in module II. The higher number of CHWs providing incorrect responses was driven largely from incorrect responses to a specific question in unit II of module II for which an average of 49% of CHWs answered incorrectly (Figure 7).

Figure 7 Illustration of average response rates for unit questions in training module I and II.

From the monitoring of the responses to all of the training-related questions it was noted that around 84% of the CHWs skipped a question at some time in the system. On average the highest rate of skipping questions was observed in unit three of the vaccine training module. About half of the CHWs were not aware of having skipped a question, 25% reported that their phone did not function well or claimed that the system was repeating the question, while around 5% reported having skipped a question due to the question being too difficult to answer.

Perception of the MOTS platform

Based upon the interviews held with the CHWs participating in the pilot, it was noted that all of the CHWs reported finding the MOTS training modules as being useful. Approximately 86% of the CHWs reported that the content of the training was very easy to understand and around 14 percent reported that the content of the training was somewhat easy to understand.

When asked about the application of the learning made from the platform, 91% of the CHWs reported having shared the learnings they had gained on the module about vaccination to the households. When the same question was asked to the households, 81% of the households surveyed affirmed that a CHW visited and provided counselling about vaccination. Further still 69% CHWs reported having shared information about outbreak response with members of the household, however only 28% of the households could affirm this counselling session happening (Figure 8).

Figure 8 Illustration of feedback from Households and CHWs on sharing their learnings during counselling. CHWs, community health workers.

Lessons learned from the Project Feedback Conference

Discussions with the participating CHWs generated a number of recommendations from the pilot of the platform that could be considered for subsequent further implementation of MOTS in other areas in Sierra Leone. The key recommendations from the focus groups were related to the ability of remotely located CHWs to have their phones charged and the possibility to incentivize CHWs for participating in these trainings. At the supervisor level, those who monitored CHWs progress using the MOTS-generated reports via their phones mentioned that this sometimes created issues when they had insufficient phone credit to facilitate their consultation of the reports. Further comment was given to the fact that the Mende local language messages were more difficult to understand as the participants spoke a different dialect of this language. Other positive observations were made regarding the degree of engagement from CHWs that the training induced and also the utility of the visual aid that was provided to accompany the platform and how they helped enable successful training.

Discussion

Sierra Leone represents a country rebuilding its health infrastructure after a bloody civil war and the largest Ebola outbreak to date in its history. After the Ebola outbreak, CHWs were recognized for their importance of extending critical health services to communities and the government has begun professionalizing and bolstering the country’s health system with the presence of approximately 15,000 CHWs. Despite this critical extension of health services, it is likely that only about 40% of these health services are being extended to rural communities due to the location of most formal health services. Ninety-one percent of CHWs live within five kilometers of the lowest level health units. CHWs were estimated to extend an additional 10 percent to geographic access beyond what is covered within 5 Km of a functioning PHU (7).

Even in a country that is so challenged like Sierra Leone, technology makes its way even to the most remote locations and reports dating from 2016 show that at least 83% of the population have access to a mobile phone and even in rural communities this percentage can be as high as 78% (8). This phenomenon is even more prevalent in the younger generations for whom mobile phone access and usage patterns suggest that more young people could be reached via mobile phones than other media (9). Given the demographics of the CHWs observed in this pilot, mobile phones would therefore seem to form an ideal vehicle for refreshing key training messages for the youthful CHW community.

The MOTS pilot that was executed in Bo district, Sierra Leone was set up to address the key questions as to whether the selected IVR technology approach would be well accepted by CHWs and other key stakeholders and also whether knowledge refreshed or acquired by CHWs using MOTS. Based on the data gathered, MOTS seems to be very well received by the CHWs, their supervisors and other managerial layers within the MoHS involved in the pilot thereby addressing the primary objective of the pilot. This finding is consistent with other research that has shown that mobile technology interventions are found to be feasible and acceptable interventions for CHWs, their supervisors and support staff (10).

While there was knowledge acquisition demonstrated for participation in both modules, fewer than 50 percent of the CHWs who participated in the quizzes were able to answer at least 80 percent of the questions correctly. Also, the level of knowledge acquisition was less for the second training module (Ebola) disease surveillance & outbreak response, which raises the question whether there was any form of fatigue that influenced the response rates of CHWs by the time they had completed both modules. The lower level of knowledge acquisition reported for the (Ebola) Disease Surveillance & Outbreak Response training module may have also been influenced by more difficult training content or ambiguous formulation of questions posed in the IVR quizzes. Research comparing methodologies for collecting data is still nascent (for example, face-to-face surveys versus IVR surveys), but is growing and there are no conclusions yet as to the effectiveness of one method over another (11). However, the effectiveness of the IVR-based quizzes deserve further inquiry. For example, a study conducted comparing IVR versus computer-assisted telephone interviews found the need for careful design of survey questions to ensure clarity and to minimize cognitive burden for respondents, many of whom may not have prior experience in taking automated surveys (12).

Broadly, the literature on the use of mobile technology for use by CHWs is growing and suggests that mobile tools help CHWs to improve the quality of care, efficiency of care, and monitoring (10). Research assessing the effectiveness of IVR-based training, specifically, with CHWS in Senegal found that knowledge change was not consistent for all content assessed using an IVR-based system, but that overall, knowledge improved as a result of IVR-based and SMS-based refresher trainings on family planning (13). Two studies conducted with CHWs in India found IVR-based training regarding maternal and child health resulted in knowledge gains and improved confidence among CHWs (14) and mixed findings regarding the onward impacts of the CHW-beneficiary interactions and behavior change (15).

During the initial stages of the pilot, several face-to-face interactions with the CHWs and their supervisors were needed in order to explain the use of the platform or to subsequently address questions the CHWs had when back in the field. This begs the question whether with a larger scale implementation this interaction can be removed or limited to allow a more lean delivery of the training platform that no longer requires face-to-face explanations. More learning on what the minimum amount of time spent together with the CHWs for initial explanation of the MOTS platform will be gleaned from the subsequent roll-out of the platform in another district.

The use of mobile technology to deliver training in short voice files does not obviate the need for classroom training (13,16). The MOTS platform is seen as a way to augment existing face-to-face training efforts being made by the MoHS in Sierra Leone. This offers the advantages that remotely-located CHWs do not need to travel to a specific location to receive refresher training and also that there is flexibility as to when CHWs can fit the training into their personal schedule. The fact that CHWs can follow such training at home may help remove possible logistical and or cost barriers and therefore can contribute to a better schooled healthcare provider population. A further advantage of this training approach is that in outbreak situations when training large groups of people together may need to be avoided to prevent transmission, remote training may offer a way to still reach CHWs with important training content.

Research into the optimal cognitive approaches to maximize learning impact underscores the need for repetition (16). Repetition is considered one of the most powerful variables affecting memory and retention of learning content for the long-term. In countries that are already stretched to deliver adequate face-to-face classroom training this technology solution provides the opportunity for repetition of key messages and serves as a way to equally reach those more remotely located.

Delivery of training via mobile phones despite having many advantages also has the limitation of telecom coverage in certain settings. Nevertheless, the growth and reported plans from various mobile network operators in Sierra Leone would make one optimistic that this possible boundary would quickly be overcome as companies compete to implement fiber-optic cable across districts in Sierra Leone. The accessibility to electricity to charge telephones is also a specific challenge for working in this or similar environments where the electrical grid is not reaching all remote areas or is not operating in a constant fashion. The recommendation of foreseeing solar-powered charging banks to enable CHWs to always have a charged telephone is seen as limited investment that local authorities may need to make in infrastructure to ensure accessibility of the CHWs at all times.

Further implementation of the platform is foreseen in collaboration with the MoHS in Kambia district of Sierra Leone. This implementation will involve more than 800 CHWs spread over 68 PHUs and 7 Chiefdoms. The training content for the (Ebola) Disease Surveillance & Outbreak Response module will be refined to ensure a better clarity of questions posed and also to enhance the probability that a greater degree of knowledge can be acquired or refreshed. This larger scale implementation will, as mentioned, be used to observe the effect of limiting the face-to-face time spent to explain the use of MOTS as well as the possible impact of the order in which trainings are delivered. This implementation also offers an opportunity to assess the cost component of a full-scale national roll-out should the local authorities wish to adopt the MOTS platform as an integral part of their health strengthening strategy. Research suggests that strong government ownership of this process is needed to scale-up interventions such as MOTS (10,17).

In parallel to the further implementation in Kambia district, discussions are ongoing with the Ministry of Health & Sanitation in Sierra Leone to assess what is required to transition the ownership of the platform to the local authorities and ensure a sustained future for this type of training platform as part of the national training toolkit.

Representatives of the MoHS made a field visit to observe the use of the MOTS platform in the field by CHWs and their supervisors. Their recommendations for improvements to the platform going forward were well aligned with the observations made by the CHWs themselves in either interviews or focus group discussions. The suggested items for improvement were the provision of a solar-powered charging banks to ensure that the CHWs telephones were readily charged. Further for CHW supervisors it was suggested that a solution be sought to ensure the supervisors can access progress reports either by foreseeing a reliable internet access for them to download data or finding another solution as to how they can access the progress reports. The availability of the training modules to CHWs beyond the time-window of the pilot was also recommended by the ministerial representatives.

About one percent of the CHWs were in a higher age group (<70 years old). A recommendation was given to focus on specific training for this age-group on how to navigate the telephone keyboard as older CHWs may not always be as proficient in this as younger colleagues. The MOTS platform was accessed by CHWs by flashing the system i.e., calling into a number on a no-charge basis and the system calling back the CHW to start the training. It was suggested that consideration be given to providing the mobile network company with the telephone numbers of the CHWs participating and simply pushing the trainings centrally in order to alleviate the need of the CHWs to have a minimum credit on their phones to allow them to ‘flash the system’.

Study limitations

This study relies on a simple pre- and post-test conducted using the IVR system quizzes with the 125 CHWs that participated in the MOTS pilot and does not include a counter-factual. As was indicated for the pre- and post-test IVR quizzes, many questions were skipped, which dilutes the understanding of actual knowledge change and may risk over- or underestimating the knowledge change that occurred. Further research is being conducted in a new district where the order of the modules is being varied to assess possible participant fatigue and to compare the results with a comparison group. This current research contributes to the literature in two ways: (I) it adds to the lessons regarding the use of IVR for refresher trainings of CHWs and (II) contributes to lessons regarding the use of IVR to assessing knowledge change of CHWs that participate in IVR trainings.

Conclusions

Based upon the experience of the pilot in Bo district, the MOTS system can readily be positioned alongside classroom training approaches in order to provide refresher training reiterating key messages delivered already to CHWs. The MOTS design that was informed by CHW feedback & the National Digital Health Strategy makes it accessible to integrate with existing ministerial systems and the open source aspect of the code removes major investment obstacles that could otherwise block adoption of this platform by authorities in a sustainable way. This IVR technology should therefore be considered when planning delivery of training to CHWs and can be positioned as a vehicle by which repetitive aspects of important training content can be reinforced without the need for additional classroom presence of the CHW community.

Acknowledgments

The EBODAC project team would like to acknowledge the contribution of the community health workers in Bo district, their supervisors, the Bo District Health Management Team and World Vision Sierra Leone research team. The management teams at the Ministry of Health & Sanitation in Sierra Leone and the community of Bo district that participated in this MOTS pilot and made the generation of these findings possible.

Funding: This EBODAC project has received funding from the Innovative Medicines Initiative 2 (IMI2: IMI’s website: www.imi.europa.eu) Joint Undertaking under grant agreement EBODAC (grant nr. 115847). This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and European Federation of Pharmaceutical Industries and Associations (EFPIA).

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

References

1. Centers for Disease Control and Prevention (CDC), 2016. Ebola outbreak in West Africa – Case Counts. Available online: https://www.cdc.gov/vhf/ebola/history/2014-2016-outbreak/case-counts.html
2. Ebola Vaccine Deployment, Acceptance and Compliance (EBODAC) webpages. Available online: https://www.ebovac.org/ebodac/
3. Pietro Dionisio. Unsung heroes: Community Health Worker Lessons in Sierra Leone post-Ebola. International Health Policies, April 2017. Available online: http://www.internationalhealthpolicies.org/unsung-heroes-community-health-worker-lessons-in-sierra-leone-post-ebola/
4. Sierra Leone National CHW policy. Available online: https://www.advancingpartners.org/sites/default/files/sites/default/files/resources/sl_national_chw_policy_2016-2020_508.pdf
5. Bo District in Sierra Leone. Available online: https://www.citypopulation.de/php/sierraleone-admin.php?adm1id=31
6. Assessment of the Readiness of Community Health Workers to participate in a Mobile Training and Support Services Innovation Results of a functionality Assessment in Bo District, Sierra Leone 2018. Available online: http://www.imedpub.com/articles/assessment-of-the-readiness-of-community-health-workers-to-participate-in-amobile-training-and-support-services-innovation-results.pdf
7. Human Resources for Health Country Profile - Sierra Leone Country Profile. Available online: https://www.afro.who.int/sites/default/files/2017-05/hrhprofile16.pdf
8. Communication in Sierra Leone: An Analysis of media and mobile audiences. BBC Media Action report May 2016. Available online: http://downloads.bbc.co.uk/rmhttp/mediaaction/pdf/research/mobile-media-landscape-sierra-leone-report.pdf
9. The Mobile Economy West Africa 2019. GSM Association. Available online: https://www.gsma.com/r/mobileeconomy/west-africa/
10. Braun R, Catalani C, Wimbush J, et al. Community Health Workers and Mobile Technology: A Systematic Review of the Literature. PLoS One 2013;8:e65772. [Crossref] [PubMed]
11. Greenleaf AR, Gibson DG, Khattar C, et al. Building the Evidence Base for Remote Data Collection in Low- and Middle-Income Countries: Comparing Reliability and Accuracy Across Survey Modalities. J Med Internet Res 2017;19:e140. [Crossref]
12. Pariyo GW, Greenleaf AR, Gibson DG, et al. Does mobile phone survey method matter? Reliability of computer-assisted telephone interviews and interactive voice response non-communicable diseases risk factor surveys in low and middle income countries. PLoS One 2019;14:e0214450. [Crossref] [PubMed]
13. Diedhiou A, Gilroy K, Cox CM, et al. Successful mLearning Pilot in Senegal: Delivering Family Planning Refresher Training Using Interactive Voice Response and SMS. Glob Health Sci Pract 2015;3:305-21. [Crossref] [PubMed]
14. Yadav D, et al. 2017. Sangoshthi: Empowering Community Health Workers through Peer Learning in Rural India. International World Wide Web Conference Committee (IW3C2). Available online: http://papers.www2017.com.au.s3-website-ap-southeast-2.amazonaws.com/proceedings/p499.pdf
15. Borkum E. 2014. Midline findings from the evaluation of the Ananya program in Bihar. Mathematica Policy Research. Available online: https://www.mathematica-mpr.com/download-media?MediaItemId=%7BED59F39A-738C-4DAE-81A8-914CABC611D6%7D
16. Bluestone J, Johnson P, Fullerton J, et al. Effective in-service training design and delivery: evidence from an integrative literature review. Hum Resour Health 2013;11:51. [Crossref] [PubMed]
17. Ngabo F, Nguimfack J, Nwaigwe F, et al. Designing and Implementing an Innovative SMS-based alert system (RapidSMS-MCH) to monitor pregnancy and reduce maternal and child deaths in Rwanda. Pan Afr Med J 2012;13:31. [PubMed]