Health Education To Villages




Home Programmes Resources India Partners Site Map About Us Contact Us
 

Centers for Disease Control and Prevention
 CDC Home Search Health Topics A-Z
Foodborne and Diarrheal Diseases BranchLink Online to Foodborne and Diarrheal Diseases BranchCDC MenuSafe Water System Menu

Safe Water System Manual


11.0
Plan monitoring and evaluation of the project

Tasks:

For monitoring

  • Identify activities/indicators/outcome measures to be monitored

  • Decide how the findings will be acted on
  • Identify sources for monitoring data and data collection methods
  • Schedule monitoring
  • Design and pretest simple forms and questionnaires for recording information

For evaluation

  • Review project objectives and relevant project activities in terms of expected effects

  • Identify indicators/outcome measures to evaluate
  • Determine sources of data for evaluation and data collection methods
  • Plan for data gathering including schedule and staff

 

Monitoring and evaluation are crucial to effective management of a Safe Water System project. There are many examples where information from monitoring or evaluation led to a significant change in a project that, one can see in retrospect, was essential to success. If a problem had not been identified, or not been identified until later, the project would have failed. In Pakistan, for example, ongoing monitoring of a project identified a problem with vessel breakage 6 - 12 months after distribution. On analysis the problem was due to ultraviolet light degradation of the plastic. The solution was to add UV light absorbers to the plastic of future vessels. In Madagascar, the project expanded from the city into a rural region affected by a cyclone. The only water source was a river with very turbid water. The dose of disinfectant recommended for clear piped water in the city was inadequate for the river water in the rural project. The solution was to double the dose.

Monitoring requires ongoing data collection during project implementation. Purposes of monitoring include:

• measuring progress of activities during implementation, using indicators, which usually relate to quality or quantity and a particular timeframe.

• highlighting which activities are being carried out well and which less well.

• providing information during implementation about specific problems and aspects that need modification.

• enabling managers to decide about allocation of resources and to identify training and supervision needs.

Evaluation requires data collection before and after a given period of project implementation. Purposes of evaluation include:

• assessing whether the objectives have been achieved.

• looking at overall strengths and weaknesses.

• guiding design of future phases or follow-up projects.

To plan for monitoring and evaluation, specify the information that will be needed, how it will be used for decision making and how progress and impact will be measured. Plans for monitoring and evaluation should be developed at the same time and integrated with plans for the whole project. At the beginning of the planning process, decide how monitoring and evaluation data will be acted on. Ensure that each piece of data collected has a purpose so that monitoring and evaluation is a meaningful practice that advances the project's goals and objectives.

 

To plan for monitoring:

11.1 Identify the activities/indicators/outcome measures to be monitored

First review the project objectives (these were specified in step 2.0) and the activities planned (in steps 9.0, 10.0) to achieve the objectives. Monitoring should allow the project to determine what activities are occurring, not occurring, or not working out as planned, so that corrections can be made along the way. The project may monitor an activity, or some indicator of the activity. For example, one activity to monitor is:

Place disinfectant and vessels in 60 retail shops and demonstrate consistent supply

This activity could be monitored by reviewing records of shipments of disinfectant to all 60 shops. Or an indicator could be monitored, such as presence of disinfectant on the shelves during a survey of a sample of retail shops. Monitoring will need to be more intensive in the early stages of the project. Once the project is established and running well, monitoring frequency can be reduced.

Limit the items to be monitored to a manageable number that will provide the most useful information for the pilot project, and that will not require excessive personnel time and project money.

 

11.2 Decide how the findings will be acted on

It is important to assure that only useful data is collected so effort is not expended on activities that do not contribute to the project. The best way to do this is to think through how each piece of monitoring data will be acted upon. For example,

• If production of disinfectant solution is insufficient to meet demand, then the project can purchase new hypochlorite generating machines, or if a company is making the solution, that company can produce more.

• If the population is not purchasing the disinfectant solution because the price is too high, the project will have to reconsider how much to charge for disinfectant.

• If the solution is not being purchased because of taste, then more education and behavior change approaches will be needed.

• If mothers with a lower educational or income level are not purchasing disinfectant, the project can undertake an educational/behavior change campaign targeting this group.

• If vessels are not being purchased because of the price, then the price will have to be changed, or a locally-produced, cheaper vessel can be recommended.

 

11.3 Identify sources for monitoring data and data collection methods

To monitor some indicators, new data collection systems may need to be established, whereas, for others, existing data sources will be sufficient. For example, systems for recording sales of vessels and disinfectant at shops or other outlets may need to be established. Alternatively, it may be simple to track invoices which are already collected by businesses for all of their sales. Specify where monitoring will be done, that is, in the whole project area or in a sample of outlets. Keep surveys limited in scope as they are labor intensive and relatively expensive. Surveying a small sample of shops, or doing a focus group, may be sufficient as a simple monitoring check of whether certain activities are getting done, or whether products are available in the target area.

Decide what methods of data collection will be used to measure the selected indicators. Possible methods include:

• Routine reports, such as

­ Records from chlorine production site about volume produced and distributed (see Annex C)

­ Reports from sales outlets of bottles sold

­ Overall sales by community and region

• Supervisory visits to health facilities that are promoting and selling the products

• Survey of outlets in target area (can include interview with staff, examination of records of sales, observation of sales behaviors, inventory of stock)

On the next page is an example of planning for monitoring. The project objectives and activities are listed, along with the data source and method of data collection for monitoring each activity.

 

Figure 17: Example: Plan for Monitoring Objectives:

Objectives:
1.1 - Sell 20,000 bottles of disinfectant in first 3 months
1.2 - Sell 1,000 water storage vessels in first 3 months

Activities to monitorData SourceMethod of data collection

Production and sales

1. Produce 1500 liters of solution per month

2. Produce 100% of batches of solution with free chlorine > 0.5%

Disinfectant production records including
concentration of batches and volumes bottled
Review of production records;

Visit to validate concentration testing
3. Train 30 community-based distributors (such as community volunteers) Records of training sessions conductedand attendees

Observation of training session
Visit to trainers / administrative office to review training records

Observation of training session to confirm agenda

4. Place disinfectant and vessels in 60 retail shops and demonstrate consistent supply

5. Place disinfectant or vessels in 10 health facilities

Survey of outlets and health facilities in target areaBi-weekly visits to outlets and health facilities to inventory stock on shelves

6. Sell at least 5000 bottles per month

7. Sell at least 300 vessels per month

Sales reportsReview sales reports

Objectives:

2.1 - 70% of target population will recognize the brand name of the Safe Water System products (vessel and disinfectant) after 6 months
2.2 - 30% of households will report use of approved water storage vessel and disinfectant after 6 months
2.3 - 25% of households will have knowledge of correct dose of disinfectant after 6 months
2.4 - 25% of households will have observed safe water storage practices after 6 months
2.5 - 10% of households will have measurable residual free chlorine levels >0.2 mg/liter after 6 months
2.6 - 10% of households will have no detectable E. coli colonies in stored water

Activities to monitorData SourceMethod of data collection

Education and promotion

1. Design and produce 20,000 information brochures

Invoices from printing companyReview invoices
2. Train 10 community health workers to deliver education messages in each of 3 communities Training recordsObservation of training sessions
Review of records
3. Hold informational meeting in 10 communities per monthMeeting minutesReview of records
4. Design and broadcast one advertisement on the radio 3 times per day Review of advertisements drafted/ produced

Review broadcast schedule

Broadcasts
Meeting with designer of advertisements and written broadcast schedule

Listen for scheduled broadcasts
5. Conduct educational event in 4 schools per month Training plans and scheduleReview plans and records of events conducted
6. Produce video and show it to 3 communities per week Drafts of video

Report of communities visited with video
Review draft and final video

Video projection truck reports of communities visited
7. Observe health facility staff providing education to mothers once per week Survey of health facilities in target areaBi-weekly visits to outlets and health facilities to observe staff

Community mobilization

8. Establish neighborhood committees in 3 communities in first 3 months

9. Have 3 committees work throughparticipatory process in 3 months

10. Have 3 communities organize themselves for the project in first 3 months

Meeting minutes

Review minutes

Observe meetings

Motivational interviewing

11. Train 10 trainers in first 3 months

12. Each trainer trains 5 additional trainers in 3 months

13. Each of 50 trainers trains 10 volunteers in 3 months

Training reports

Review reports

Observe training

14. Each volunteer conducts motivational interviewing intervention in 10 community households in 3 month period Regular meetings with volunteers

Reports from volunteers

Accompany volunteers on some visits

 

11.4 Schedule monitoring

Decide on the timing of monitoring activities and draw up a schedule. Some activities or indicators may require weekly or monthly monitoring, whereas others may only need to be measured once or twice during a 6 - 12 month pilot project. Some are important to check early in the project, so that problems can be quickly identified and solved. For example, if there are problems producing enough disinfectant to supply outlets, this problem must be quickly solved. If it is found that vessels are available but people are not buying them, this problem should be addressed quickly. Plan to begin data collection while the pilot project activities are implemented.

Identify staff who will conduct monitoring and who will analyze and interpret the data and schedule their time.

 

11.5 Design and pre-test simple forms and questionnaires for recording information

For example, design forms to collect information about sales of vessels and disinfectant at outlets. When designing data collection, be sure to include all the information needed to monitor the selected indicators, but do not include extra information that is "nice to know" but will not be used for decision making.

 

To plan for evaluation:

11.6 Review project objectives and relevant project activities in terms of expected effects

Review the reasons for evaluating the project and review the project objectives. Then describe the project:

• target population

• project activities

• responsibilities of project staff

• resources available to project (persons, transportation, data managers/analyzers, money)

Match project objectives with project activities to be sure enough activities will be done to meet the objectives in the proposed time frame. This will prevent undertaking evaluation prematurely, when there is little or no chance of measuring an impact. For example, to measure the health impact of the project, 20 percent of the population will need to be using the intervention. Until the project has reached this level of participation, a health impact evaluation will be premature.

 

11.7 Identify indicators/outcome measures to evaluate

Specify the indicators/outcome measures based on what is important to know to evaluate achievement of project objectives, strengths and weaknesses of the pilot, and to plan for future activities and expansion. Limit the items to be evaluated to a manageable number that will provide the most useful information and that will enable you to stay within budget and personnel limitations of the project.

Evaluation of the behavior of the target population and use of the products is essential. If use of the products is less than expected, or declining, the project must figure out the reasons and make adjustments. Behavior change strategies must be designed, implemented and modified as needed because these are the key to an effective project. Repeated project evaluations over time will permit personnel to determine whether behavior change is increasing or decreasing in the population. For the project to succeed new behaviors must be sustained.

 

11.8 Determine sources of data for evaluation and data collection methods

Possible sources include:

• interviews with members of target populations

• disease registries in health facilities serving target populations

• accumulation of monitoring results

Possible methods include:

• community surveys (baseline and follow-up), which can include interviewing family members in their homes, observing certain practices, and/or testing samples of water stored in the home for chlorine residuals or microbiologic quality

• surveys of health facilities and sales outlets, which can include interviewing staff, observing sales and education behaviors, and checking stock

• exit interviews with families attending health facilities or families purchasing Safe Water System products

• focus group interview of sample of target population (to assess people's perceptions of product acceptability, taste, cost and to probe for potential barriers to utilization, such as cultural factors, education, other priorities)

• review of monitoring results

• special studies and surveys that can assist in understanding specific operational issues, for example, case control studies of patients visiting health center with diarrhea and their well neighbors

• active diarrhea surveillance by home visits (health impact)

If there is a local laboratory that assesses microbiologic quality of water, the project may choose to assess water quality. However, these tests can be expensive. Measurement of free chlorine residuals is a reasonable indicator of microbiologic quality, since in the presence of adequate free chlorine residuals, it is much less likely that E. coli are present in the water.

The table on the next page shows a plan for evaluation. For each objective, it shows indicators to evaluate, data sources and data collection methods.

 

Figure 18: Example: Plan for Evaluation

ObjectiveIndicatorData SourceMethod of Data Collection

1. Increasing access to the intervention (hardware)

Number of bottles soldSales recordsSales recordsReview sales recordsReview sales records

1.1 - Sell 20,000 bottles of disinfectant in first 3 months

1.2 - Sell 1,000 water storage vessels in first 3 months

Number of vessels sold

2. Changing water treatment and storage behaviors   
2.1 - 70% of target population will recognize the brand name of the Safe Water System products (vessel and disinfectant) after 6 months% recognition of brand nameInterviews with target populationBaseline and follow-up surveys of random sample of target population
2.2 - 30% of households will report use of approved water storage vessel and disinfectant after 6 months% of households reporting use

2.3 - 25% of households will have knowledge of correct dose of disinfectant after 6 months

% of households able to demonstrate correct dose

2.4 - 25% of households will have observed safe water storage practices after 6 months% of households with stored water observed in recommended containerHome visits to observe water storage practicesBaseline and follow-up surveys of random sample of target population
2.5 - 10% of households will have measurable residual free chlorine levels > 0.2 mg/liter after 6 months.% of households with free chlorine residual >0.2 mg/literWater stored in householdsHome visits to random sample of population to test stored water at baseline and then after 6 months of implementation
2.6 - 10% of households will have zero E. Coli colonies in stored water% of households with zero detectable E. Coli colonies in stored water
3. Improving health% decreased risk of diarrhea in intervention households compared to control householdsInterviews with patients or caregiversActive diarrhea surveillance: periodic (weekly, biweekly) home visits to obtain information about diarrhea episodes. Need comparison group -- could be non-users of intervention or selected control group. Obtain baseline diarrhea data, and then data following implementation of intervention
3.1 - Reduce diarrhea rates in target population by 20%.
4. - Achieving satisfaction% of households indicating satisfaction with productsInterviews with households in target populationSurvey of random sample of target population
Focus group interviews
4.1 - 80% of households in target population will report satisfaction with products

 

11.9 Plan for data gathering including schedule and staff

• Select a data gathering method.

• Consider the purpose of the evaluation, the anticipated start of project activities, and time required for intended outcomes to occur. Then specify when to collect baseline data (prior to project implementation) and when to collect evaluation data (after an appropriate interval).

• When estimating time and other resources required for the data collection, also consider:

- the number of project participants (e.g., homes or outlets to be surveyed), distances between these

- the willingness of participants to provide data, the difficulty and time required for data collection at each house (e.g., testing water samples, interviewing family members, observation of water handling practices)

• Design and pretest simple forms and questionnaires for data collection.

• Determine who will collect, analyze, and interpret evaluation data.

• Also determine who will be responsible for writing report. Without a written report, the evaluation will not be in a useful form. The report is essential for progress reports to donor agencies and can provide justification for future funding.

• Set timeline for data collection, analysis, interpretation, and report writing.

Home

Centers for Disease Control and Prevention
National Center for Infectious Diseases
Division of Bacterial and Mycotic Diseases
Foodborne and Diarrheal Diseases Branch

 

 The Mother and Child Health and Education Trust Programmes | Resources | India | Partners | Site Map | About Us | Contact Us top of page

Feedback Form