Step 8: Develop M&E Plan and Tools and Implement

Objectives

  • Develop Monitoring and Evaluation (M&E) plan
  • Develop and print M&E tools
  • Conduct monitoring activities on an ongoing basis
  • Plan and conduct evaluation activities

Outputs

  • M&E plan
  • Monitoring activities undertaken:
    • Routine supervision
    • Health management information system (HMIS) tracking
    • Logistics tracking
    • Commodity management assessments
  • Evaluation tools for ITN access assessments
  • Cost analysis

  • Convene coordination committee to draft M&E plan and tools.
  • Have wider stakeholders review plan.
  • Finalize and widely distribute plan.
  • Ensure evaluation tools are printed and in place, in a timely manner.
  • Ensure on-going collection of data for cost analysis.
  • Ensure on-going monitoring activities, including routine supervision, HMIS tracking, logistics tracking, and commodity management assessments.

  • Planning for M&E activities will vary, depending on whether a pilot implementation is planned. If implementation of a new channel has been undertaken at a small pilot scale, it will probably be appropriate to have more detailed evaluation activities.
  • Collection of accurate information on indicators of program progress is a key element for managing and improving program quality. Most often, donor agencies will require it.
  • Monitoring data requirements should focus on need-to-know data (based on defined indicators) and not include nice-to-know data. Simplification of data requirements is important to improve accuracy, completeness, and timeliness of information collected.
  • Ideally, monitoring data shows the flow through standard systems. Report health-facility distributions through the HMIS channels, for example, where possible (though see channel specific guidance on this issue for more information). School-based distributions will not likely have existing health reporting structure within which to work; therefore, it will need parallel reporting systems. Measurable guidance can be drawn from campaign style monitoring (see the Alliance for Malaria Prevention [AMP] toolkit for guidance). HMIS tracking is discussed in Step 5.
  • Activities that may be useful to routinely monitor and evaluate a CD ITN distribution include—
    • Logistics tracking by collating the ITN audit trail documentation (use logistics tracking documentation, as discussed in Step 4).
    • Supervising activities against specific criteria, including adherence to communications activities and quality of training. Collate the supervision reports and findings to inform the planning for subsequent rounds. See Step 5.
    • Commodity management assessment—to review accountability. This guide provides more information.
    • Collation and analysis of distribution data to determine:
      • Number of ITNs delivered.
      • Efficiency of the distribution (i.e., what proportion of potential recipients are recorded as receiving an ITN).
      • For community distribution, proportion of coupons redeemed for ITNs.
    • Useful indicators to inform planning include the proportion of—
      • Targeted individuals who received ITNs.
      • ITNs given out that successfully reached the household.
      • ITNs retained by the households.
      • ITNs given to other households.
      • ITNs used the previous night, by whom, and by how many people.
    • Community surveys to estimate—
      • The proportion of ITNs received that were taken to their households.
      • The proportion of ITNs received that someone slept under the previous night. To understand the context, data on other ITNs and the household structure will also be needed.
      • Access of ITNs at population (access) and household levels to provide evidence that universal access is being maintained, with sub-analysis to identify the contribution of various channels.
      • Proportions of eligible households who received the ITN through a given channel.
      • Identify characteristics of households who have not or did not receive nets through any channel.
      • To explore community members’ perceptions and experience with the program.
  • Monitoring activities can be time consuming and costly, and there is no clear evidence as to what scale of monitoring improves performance or better accountability. Sometimes, it is alright to use nationally representative survey data sets to answer some of the outcome questions. Alternative options can also be considered; for example, telephone monitoring has the potential to provide as good and more efficient logistics management and accountability monitoring in some contexts. However, give careful consideration to the benefits it would bring and the potential to add parallel or burdensome activities.
  • A key challenge of evaluating the performance of continuous distribution channels is that, in ideal circumstances, countries will use multiple delivery channels, including mass campaign distributions, as well as various CD channels. Therefore, it may be difficult to separately evaluate the impact of ITN delivery via specific channels. However, the source of each net is now a standard question in Demographic and Health Surveys (DHS) and Malaria Indicator Surveys (MIS) will facilitate such an analysis in the future. An analysis plan and Stata do-file are available for this, to help evaluate:
    • % of pregnant women attending ANC who received an ITN at an ANC
    • % of children attending EPI who received an ITN at the EPI clinic
    • % households with school pupils of target age who received an ITN
    • % of ITNs received from different sources
  • It is recommended that the durability of ITNs be monitored whenever ITN distributions are planned at scale. This includes measuring the insecticidal and the physical durability of the ITNs, as well as the attrition rates. Various guidance is available to support durability monitoring.
  • Collect on-going cost data to support a cost analysis as part of the evaluation.

  • Where possible, collect monitoring data within the HMIS system. However, it may be challenging if HMIS functionality and accessibility does not make data available in a timely and sufficiently complete way for donor reporting. Avoid setting up duplicate systems (e.g., sending staff out to collect data from distribution points in time for a report), but focus on ensuring the available program resources supporting the flow of data through the HMIS.
  • NMCPs and their partners will need to work closely with the MOH leadership to review the potential of the HMIS to collect routine data on ITN delivery via health services. This review requires a national-level dialog, led by the MOH, and including the NMCP, reproductive health (RH), EPI, MCH, and other divisions involved in overall planning, as well as the MOH division in charge of the HMIS. During these discussions, it is important to review the level of functionality of the current system, options for including routine ITN distribution information in the existing HMIS, and the possibility of establishing temporary parallel structures while working to strengthen the national HMIS. In one country, a column for ITNs distributed was added to the ANC register, but the printing of the new registers and summary forms were delayed. In this instance, health workers were advised to draw a line (to create such a column) in their current registers. In another country, these tallies were collected and reported in support supervision visit reports. Donor agencies and other partners should be present to inform and to learn from the discussion. However, the MOH should make the final decision.

Important questions to be answered during this analysis are—

  • What system is currently used for HMIS data collection and how easily can it be modified (e.g., DHIS2)?
  • What is the current level of reporting from health facilities/districts to the national level?
  • What is the level of accuracy of reporting received (results from data audits)?
  • What are the time lines for receiving information from the HMIS that can be relayed to program managers, donors, and other partners?
  • Do reporting lines clearly determine who receives malaria information and who is able to supervise the quality, completeness, accuracy, and promptness of malaria information?
  • Are activities underway to strengthen the HMIS? Who are the partners involved in this process? What are their recommendations?
  • The VectorWorks document, Guide to Health Facility-based Distribution of Insecticide-treated Nets, provides more detail on these issues.

  • Key monitoring aspects are how well eligibility criteria are being assessed and adhered to, how many eligible community members are accessing coupons, and what are the coupon redemption rates. These aspects will be particularly key to ensure that redesign can be undertaken, if needed.
  • The following information should be reported by those involved in community distribution programs:
Supervisors Coupon distributors Redemption point storekeepers Transporters
Number of coupon books issued

Number of supervision visits made

Main outcomes of supervision visits

 Number of coupons distributed

Summary of reasons for coupon distribution

 Number of coupons redeemed

Numbers of ITNs issued

Number of days of stock-outs

Stock level

 Number of ITNs delivered to central stores, regional hubs, and redemption point stores

Percentage of orders delivered on time

Item fill rate (percent of ITNs ordered that were delivered)

Percentage of deliveries with proof-of-delivery forms on file

M&E materials: