Safe
Water System Manual
11.0
Plan monitoring and evaluation of the project
Tasks:
For
monitoring
For
evaluation
Monitoring
and evaluation are crucial to effective management of a Safe Water System project.
There are many examples where information from monitoring or evaluation led to
a significant change in a project that, one can see in retrospect, was essential
to success. If a problem had not been identified, or not been identified until
later, the project would have failed. In Pakistan, for example, ongoing
monitoring of a project identified a problem with vessel breakage 6 - 12 months
after distribution. On analysis the problem was due to ultraviolet light degradation
of the plastic. The solution was to add UV light absorbers to the plastic of future
vessels. In Madagascar, the project expanded from the city into a rural
region affected by a cyclone. The only water source was a river with very turbid
water. The dose of disinfectant recommended for clear piped water in the city
was inadequate for the river water in the rural project. The solution was to double
the dose.
Monitoring
requires ongoing data collection during project implementation. Purposes
of monitoring include:
•
measuring progress of activities during implementation, using indicators, which
usually relate to quality or quantity and a particular timeframe.
•
highlighting which activities are being carried out well and which less well.
• providing
information during implementation about specific problems and aspects that need
modification.
•
enabling managers to decide about allocation of resources and to identify training
and supervision needs.
Evaluation
requires data collection before and after a given period of project
implementation. Purposes of evaluation include:
•
assessing whether the objectives have been achieved.
•
looking at overall strengths and weaknesses.
•
guiding design of future phases or follow-up projects.
To
plan for monitoring and evaluation, specify the information that will be needed,
how it will be used for decision making and how progress and impact will be measured.
Plans for monitoring and evaluation should be developed at the same time and integrated
with plans for the whole project. At the beginning of the planning process, decide
how monitoring and evaluation data will be acted on. Ensure that each piece of
data collected has a purpose so that monitoring and evaluation is a meaningful
practice that advances the project's goals and objectives.
To plan for
monitoring:
11.1
Identify the activities/indicators/outcome measures to be monitored
First review the
project objectives (these were specified in step 2.0)
and the activities planned (in steps 9.0, 10.0)
to achieve the objectives. Monitoring should allow the project to determine what
activities are occurring, not occurring, or not working out as planned, so that
corrections can be made along the way. The project may monitor an activity, or
some indicator of the activity. For example, one activity to monitor is:
•
Place disinfectant and vessels in 60 retail shops and demonstrate consistent supply
This
activity could be monitored by reviewing records of shipments of disinfectant
to all 60 shops. Or an indicator could be monitored, such as presence of disinfectant
on the shelves during a survey of a sample of retail shops. Monitoring will need
to be more intensive in the early stages of the project. Once the project is established
and running well, monitoring frequency can be reduced.
Limit
the items to be monitored to a manageable number that will provide the most useful
information for the pilot project, and that will not require excessive personnel
time and project money.
11.2
Decide how the findings will be acted on
It
is important to assure that only useful data is collected so effort is not expended
on activities that do not contribute to the project. The best way to do this is
to think through how each piece of monitoring data will be acted upon. For example,
•
If production of disinfectant solution is insufficient to meet demand, then the
project can purchase new hypochlorite generating machines, or if a company is
making the solution, that company can produce more.
•
If the population is not purchasing the disinfectant solution because the price
is too high, the project will have to reconsider how much to charge for disinfectant.
• If
the solution is not being purchased because of taste, then more education and
behavior change approaches will be needed.
•
If mothers with a lower educational or income level are not purchasing disinfectant,
the project can undertake an educational/behavior change campaign targeting this
group.
•
If vessels are not being purchased because of the price, then the price will have
to be changed, or a locally-produced, cheaper vessel can be recommended.
11.3
Identify sources for monitoring data and data collection methods
To monitor some
indicators, new data collection systems may need to be established, whereas, for
others, existing data sources will be sufficient. For example, systems for recording
sales of vessels and disinfectant at shops or other outlets may need to be established.
Alternatively, it may be simple to track invoices which are already collected
by businesses for all of their sales. Specify where monitoring will be done, that
is, in the whole project area or in a sample of outlets. Keep surveys limited
in scope as they are labor intensive and relatively expensive. Surveying a small
sample of shops, or doing a focus group, may be sufficient as a simple monitoring
check of whether certain activities are getting done, or whether products are
available in the target area.
Decide
what methods of data collection will be used to measure the selected indicators.
Possible methods include:
•
Routine reports, such as
Records from chlorine production site about volume produced and distributed (see
Annex C)
Reports from sales outlets of bottles sold
Overall sales by community and region
•
Supervisory visits to health facilities that are promoting and selling the products
•
Survey of outlets in target area (can include interview with staff, examination
of records of sales, observation of sales behaviors, inventory of stock)
On the next page
is an example of planning for monitoring. The project objectives and activities
are listed, along with the data source and method of data collection for monitoring
each activity.
Figure
17: Example: Plan for Monitoring Objectives:
Objectives:
1.1
- Sell 20,000 bottles of disinfectant in first 3 months
1.2 - Sell
1,000 water storage vessels in first 3 months
| Activities
to monitor | Data
Source | Method
of data collection |
Production
and sales 1.
Produce 1500 liters of solution per month
2. Produce 100% of batches of
solution with free chlorine > 0.5% | Disinfectant
production records including
concentration of batches and volumes bottled | Review
of production records;
Visit to validate concentration testing |
| 3.
Train 30 community-based distributors (such as community volunteers) | Records
of training sessions conductedand attendees
Observation of training session | Visit
to trainers / administrative office to review training records
Observation
of training session to confirm agenda |
4.
Place disinfectant and vessels in 60 retail shops and demonstrate consistent supply
5. Place
disinfectant or vessels in 10 health facilities | Survey
of outlets and health facilities in target area | Bi-weekly
visits to outlets and health facilities to inventory stock on shelves |
6.
Sell at least 5000 bottles per month 7.
Sell at least 300 vessels per month | Sales
reports | Review
sales reports |
Objectives:
2.1
- 70% of target population will recognize the brand name of the Safe Water
System products (vessel and disinfectant) after 6 months
2.2 - 30%
of households will report use of approved water storage vessel and disinfectant
after 6 months
2.3 - 25% of households will have knowledge of correct
dose of disinfectant after 6 months
2.4 - 25% of households will have
observed safe water storage practices after 6 months
2.5 - 10% of households
will have measurable residual free chlorine levels >0.2 mg/liter after 6 months
2.6 - 10% of households will have no detectable E. coli
colonies in stored water
| Activities
to monitor | Data
Source | Method
of data collection |
Education
and promotion 1.
Design and produce 20,000 information brochures | Invoices
from printing company | Review
invoices |
| 2.
Train 10 community health workers to deliver education messages in each of 3 communities
| Training
records | Observation
of training sessions
Review of records |
| 3.
Hold informational meeting in 10 communities per month | Meeting
minutes | Review
of records |
| 4.
Design and broadcast one advertisement on the radio 3 times per day | Review
of advertisements drafted/ produced
Review broadcast schedule
Broadcasts | Meeting
with designer of advertisements and written broadcast schedule
Listen for
scheduled broadcasts |
| 5.
Conduct educational event in 4 schools per month | Training
plans and schedule | Review
plans and records of events conducted |
| 6.
Produce video and show it to 3 communities per week | Drafts
of video
Report of communities visited with video
| Review
draft and final video
Video projection truck reports of communities visited
|
| 7.
Observe health facility staff providing education to mothers once per week | Survey
of health facilities in target area | Bi-weekly
visits to outlets and health facilities to observe staff |
Community
mobilization 8.
Establish neighborhood committees in 3 communities in first 3 months 9.
Have 3 committees work throughparticipatory process in 3 months 10.
Have 3 communities organize themselves for the project in first 3 months | Meeting
minutes | Review
minutes Observe
meetings |
Motivational
interviewing 11.
Train 10 trainers in first 3 months 12.
Each trainer trains 5 additional trainers in 3 months 13.
Each of 50 trainers trains 10 volunteers in 3 months | Training
reports | Review
reports Observe
training |
| 14.
Each volunteer conducts motivational interviewing intervention in 10 community
households in 3 month period | Regular
meetings with volunteers | Reports
from volunteers Accompany
volunteers on some visits |
11.4
Schedule monitoring
Decide
on the timing of monitoring activities and draw up a schedule. Some activities
or indicators may require weekly or monthly monitoring, whereas others may only
need to be measured once or twice during a 6 - 12 month pilot project. Some are
important to check early in the project, so that problems can be quickly identified
and solved. For example, if there are problems producing enough disinfectant to
supply outlets, this problem must be quickly solved. If it is found that vessels
are available but people are not buying them, this problem should be addressed
quickly. Plan to begin data collection while the pilot project activities are
implemented.
Identify
staff who will conduct monitoring and who will analyze and interpret the data
and schedule their time.
11.5
Design and pre-test simple forms and questionnaires for recording information
For
example, design forms to collect information about sales of vessels and disinfectant
at outlets. When designing data collection, be sure to include all the information
needed to monitor the selected indicators, but do not include extra information
that is "nice to know" but will not be used for decision making.
To plan for
evaluation:
11.6
Review project objectives and relevant project activities in terms of expected
effects
Review
the reasons for evaluating the project and review the project objectives. Then
describe the project:
•
target population
•
project activities
•
responsibilities of project staff
•
resources available to project (persons, transportation, data managers/analyzers,
money)
Match
project objectives with project activities to be sure enough activities will be
done to meet the objectives in the proposed time frame. This will prevent undertaking
evaluation prematurely, when there is little or no chance of measuring an impact.
For example, to measure the health impact of the project, 20 percent of the population
will need to be using the intervention. Until the project has reached this level
of participation, a health impact evaluation will be premature.
11.7
Identify indicators/outcome measures to evaluate
Specify
the indicators/outcome measures based on what is important to know to evaluate
achievement of project objectives, strengths and weaknesses of the pilot, and
to plan for future activities and expansion. Limit the items to be evaluated to
a manageable number that will provide the most useful information and that will
enable you to stay within budget and personnel limitations of the project.
Evaluation of the
behavior of the target population and use of the products is essential. If use
of the products is less than expected, or declining, the project must figure out
the reasons and make adjustments. Behavior change strategies must be designed,
implemented and modified as needed because these are the key to an effective project.
Repeated project evaluations over time will permit personnel to determine whether
behavior change is increasing or decreasing in the population. For the project
to succeed new behaviors must be sustained.
11.8
Determine sources of data for evaluation and data collection methods
Possible
sources include:
•
interviews with members of target populations
•
disease registries in health facilities serving target populations
•
accumulation of monitoring results
Possible
methods include:
•
community surveys (baseline and follow-up), which can include interviewing family
members in their homes, observing certain practices, and/or testing samples of
water stored in the home for chlorine residuals or microbiologic quality
•
surveys of health facilities and sales outlets, which can include interviewing
staff, observing sales and education behaviors, and checking stock
•
exit interviews with families attending health facilities or families purchasing
Safe Water System products
•
focus group interview of sample of target population (to assess people's perceptions
of product acceptability, taste, cost and to probe for potential barriers to utilization,
such as cultural factors, education, other priorities)
•
review of monitoring results
•
special studies and surveys that can assist in understanding specific operational
issues, for example, case control studies of patients visiting health center with
diarrhea and their well neighbors
•
active diarrhea surveillance by home visits (health impact)
If there is a local
laboratory that assesses microbiologic quality of water, the project may choose
to assess water quality. However, these tests can be expensive. Measurement of
free chlorine residuals is a reasonable indicator of microbiologic quality, since
in the presence of adequate free chlorine residuals, it is much less likely that
E. coli are present in the water.
The
table on the next page shows a plan for evaluation. For each objective, it shows
indicators to evaluate, data sources and data collection methods.
Figure
18: Example: Plan for Evaluation
| Objective | Indicator | Data
Source | Method
of Data Collection |
1. Increasing
access to the intervention (hardware) | Number
of bottles sold | Sales
recordsSales records | Review
sales recordsReview sales records |
1.1 -
Sell 20,000 bottles of disinfectant in first 3 months 1.2
- Sell 1,000 water storage vessels in first 3 months | Number
of vessels sold |
| 2. Changing
water treatment and storage behaviors | | | |
| 2.1
-
70% of target population will recognize the brand name of the Safe Water System
products (vessel and disinfectant) after 6 months | %
recognition of brand name | Interviews
with target population | Baseline
and follow-up surveys of random sample of target population |
| 2.2 -
30% of households will report use of approved water storage vessel and disinfectant
after 6 months | %
of households reporting use |
2.3 -
25% of households will have knowledge of correct dose of disinfectant after 6
months | %
of households able to demonstrate correct dose |
| 2.4 -
25% of households will have observed safe water storage practices after 6 months | %
of households with stored water observed in recommended container | Home
visits to observe water storage practices | Baseline
and follow-up surveys of random sample of target population |
| 2.5 -
10% of households will have measurable residual free chlorine levels > 0.2
mg/liter after 6 months. | %
of households with free chlorine residual >0.2 mg/liter | Water
stored in households | Home
visits to random sample of population to test stored water at baseline and then
after 6 months of implementation |
| 2.6 -
10% of households will have zero E. Coli colonies in stored water | %
of households with zero detectable E. Coli colonies in stored water |
| 3. Improving
health | %
decreased risk of diarrhea in intervention households compared to control households | Interviews
with patients or caregivers | Active
diarrhea surveillance: periodic (weekly, biweekly) home visits to obtain information
about diarrhea episodes. Need comparison group -- could be non-users of intervention
or selected control group. Obtain baseline diarrhea data, and then data following
implementation of intervention |
| 3.1
- Reduce diarrhea rates in target population by 20%. |
| 4. -
Achieving satisfaction | %
of households indicating satisfaction with products | Interviews
with households in target population | Survey
of random sample of target population
Focus group interviews |
| 4.1 -
80% of households in target population will report satisfaction with products
|
11.9
Plan for data gathering including schedule and staff
•
Select a data gathering method.
•
Consider the purpose of the evaluation, the anticipated start of project activities,
and time required for intended outcomes to occur. Then specify when to collect
baseline data (prior to project implementation) and when to collect evaluation
data (after an appropriate interval).
•
When estimating time and other resources required for the data collection, also
consider:
-
the number of project participants (e.g., homes or outlets to be surveyed), distances
between these
-
the willingness of participants to provide data, the difficulty and time required
for data collection at each house (e.g., testing water samples, interviewing family
members, observation of water handling practices)
•
Design and pretest simple forms and questionnaires for data collection.
• Determine
who will collect, analyze, and interpret evaluation data.
•
Also determine who will be responsible for writing report. Without a written report,
the evaluation will not be in a useful form. The report is essential for progress
reports to donor agencies and can provide justification for future funding.
• Set timeline
for data collection, analysis, interpretation, and report writing.
