"People themselves are at the heart of the development process and must be the subjects, rather than the objects of their own process."
Chambers (1983)
The primary participant in a community forestry project is the community. It reaps the benefits and bears the costs. Communities and community members already intuitively, informally evaluate projects in light of their own individual and/or group objectives.
Evaluation is most commonly associated with the outsiders' (donors agencies, national project staff) perspectives. They generally want to know how the project meets agency objectives. Whether the benefits of the project justify the expenditure. While important, this is not the type of information that is of primary concern in participatory evaluations. The outcomes of participatory evaluations can however, complement and enrich external evaluations, especially if the outsiders are interested in sustainability.
Participatory Evaluation Events are project evaluations in which communities and/or beneficiaries take the lead. They are encouraged and supported to take responsibility and control in planning, carrying out and reporting the results of evaluation. Outsiders (field staff, donors, etc.) support and facilitate their efforts. The logic behind this perspective is that what the community feels are the "real" costs and benefits of the project are of the highest concern.
The above diagram indicates the place of participatory evaluation events in the PAME approach. Evaluation events take some of the information from the Assessment track (Community Problem Analysis and Participatory Baselines) and some information from the Monitoring and ongoing evaluation track. They link to the activities and objectives of the project, as the information is gathered and analysed to form the basis for decisions.
A participatory evaluation event does not necessarily mean that a final judgement is being made. Rather, these events work towards making ongoing adjustments in the lifespan of the project. What can emerge is information encouraging changes or adjustments in activities and perspectives, and/or increases in clarity of purpose.
One important aspect of participatory evaluation is that it can determine whether or not project objectives are being met while also revealing the relevance of those objectives. It can indicate the need to adapt, revise, or change old objectives.
|
FROM EXPERIENCE GOAL FREE EVALUATION If the beneficiaries have set their own objectives and been involved in problem identification, an evaluation can help the community see if their objectives have been met. However, if the community has not been truly involved in the whole process, GOAL FREE EVALUATION can be more appropriate. In a goal-free evaluation data on the project's effects and effectiveness is gathered directly. The process is therefore not constrained by the (narrow) focus of the stated goals. Goal-free evaluation lends itself to qualitative methods because it relies heavily on descriptive and direct experience with the project. The evaluators must suspend judgement regarding the project's intent and focus instead on finding out what has actually happened as a result of the project. Goal-free evaluation CAN be conducted along with goal-based evaluations if separate "evaluators" are used for each approach. Source: Patton (1987) |
The overall purpose of a participatory evaluation event is to encourage projects to stop and reflect on what has happened in the past in order to make decisions about the future. By evaluating, people learn about the things that have worked well and the things that haven't worked well. They begin to realize why things have or haven't worked. And through the process, it becomes more likely that corrective measures will be implemented because they are discovered and understood by the community.
In a participatory evaluation event, the objectives of the project, as well as the expected outputs can be examined and clarified. It may be that the objectives of the community have changed, or that the expected outputs were unrealistic. Changes and adjustments may need to be made in order to achieve desired results.
Participatory evaluation events can also be used to avert a potential crisis; they provide a forum for discussion and problem solving. For example, if one sector of the community has fenced and planted an area that neglected groups previously used, there may be serious problems. Bringing the two groups together to discuss and mediate a solution can be done by using a participatory evaluation event.
The findings of a participatory evaluation event can also be presented to decision makers outside the community, giving them access to the perceptions of the community which may be difficult to obtain through other means.
|
FROM EXPERIENCE RE-ORIENTING THE DONORS A presentation at an FAO/SIDA Forests, Trees and People consultation in Rome pointed out the benefits of participatory video evaluation in communicating information directly to information users. In this way, the "filters of self-interest" were by-passed, providing a more realistic picture of the project impacts.
|
There are many reasons why a community can decide to do a participatory evaluation, and it is important that everyone who is involved clearly understands and generally agrees upon the PURPOSE of the evaluation.
Participatory evaluation events can have the following kinds of benefits. They can:
PROVIDE an entry point for trying participatory approaches to determine whether they will be effective for future, similar projects, or subsequent phases of the project being evaluated.
ASSIST in decision-making. Participatory evaluation may help beneficiaries decide whether to continue their efforts, to add or drop a part of the project, or they should try a similar approach to achieve different goals.
|
FROM EXPERIENCE ONE MAN'S CURSE IS ANOTHER MAN'S BLESSING Evaluation is not always straightforward and should not be considered a static measure for "success". Activities have often been judged quite differently by outsiders than by the community. A road which was washed away was judged a failure by project evaluators, but the community found that the surviving road base created, for the first time, a link to the market during the rainy season. The community, therefore, viewed the activity as a success. A project in Sudan which involved community nurseries was judged "not at all successful" by an outside evaluator because it failed to achieve targeted seedling production. However, the community had decided to maintain a constant production of tree seedlings, and complemented their activities by growing vegetables. The community, who were concerned with both short-term and long-term benefits judged the project as "very successful". Source: Hoskins (1986), Davis-Case (1987) |
INCREASE the community's analytical skills and awareness of their problems, needs and priorities.
ASSESS performance. Information may be used to identify weak points within a project, and to modify activities to make the project more effective. The information can also help both insiders and outsiders recognize successful aspects of the project.
ALLOCATE resources. Participatory evaluation can reveal that certain activities are more effective than others, helping to assure that resources are channeled accordingly. New areas that should be allocated resources can be identified.
DEVELOP skills. Participatory evaluation events can reveal community skills that were undervalued, and develop skills that are needed to support the project. The communication skills of the field staff can also be enhanced through a participatory evaluation event.
JUSTIFY project inputs. Participatory evaluation events may allow an assessment of whether project benefits have warranted project costs and inputs. These events also permit the community to discuss other, unanticipated benefits and costs.
DETERMINE strategies. Evaluation may reveal that activities were not effective in meeting the objectives; more appropriate approaches or activities can be recommended.
CREATE new knowledge and understanding. Evaluations can help people share valuable lessons that they have learned from field situations.
ASSURE project staffs, researchers, government planners, and donors that any inputs they make have relevance in the community situation.
Participatory evaluation events can be done at any time during the project, or even after the project has phased out. Sometimes, participatory evaluation events are planned from the beginning. Sometimes, they can be triggered by a special problem or crisis which is being confronted by the project. Problems may become apparent because of information from a participatory monitoring and ongoing evaluation (PMoe), or problems may be obvious, such as a general lack of interest in the activities. If the problems cannot be addressed without more information, or without the formal involvement of others, a participatory evaluation event may be called for.
The most valuable asset to a successful participatory evaluation event is the field staff. If PAME has been implemented from the beginning of the project, trust and communication between field staff and the community have already been well established. If the evaluation event is an entry point for PAME, time may have to be taken to train and sensitize staff to facilitate the evaluation. An outsider who is experienced in participatory evaluation approaches may be very useful. The outsider can both train the field staff and facilitate the evaluation.
|
EXPERIENCE ONE OF THE MYTHS OF EVALUATION It is a myth that only an "evaluation specialist" can do a good objective evaluation. An analyticalapproach, good practical experience and a broad, objective outlook can be found in most communities,and in project field staff. This is not to say that experienced evaluation experts cannot be helpful, particularly at the design stage, but look out for the expert who engages in "methodological overkill" and whose approach is so narrow and number oriented that it gives an incomplete and lopsided picture. |
Participatory evaluation events do not necessarily have to take more time than other types of evaluations. They are sometimes longer, sometimes shorter, depending on the subjects covered and the amount of participation. General experience seems to suggest that when participation is first introduced it can be time consuming. In light of this, the more participatory the evaluation is, the more time it takes. But, when communities are working in a participatory manner, a participatory evaluation can be one of the most effective, efficient approaches.
Carefully preparation and planning can make a participatory evaluation event easier for everybody.
Preparing first entails deciding who will be included in conducting the evaluation, and who will want to know the results of the evaluation. These decisions, made by the beneficiaries, may result in inclusion of the whole community, only the beneficiaries, or other groups (such as nomadic herders) who have an interest in, or are affected by the project. If a large number of people are interested in being included in the evaluation, it may mean that an evaluation team has to be chosen to take responsibility for organizing the information gathering. The larger group can however decide which information it needs. It can also help analyse that information.
A participatory evaluation event includes several important steps. These steps do not necessarily have to be taken in the order given, but each step is important to a successful evaluation.
A. DECIDE REASONS FOR THE EVALUATION (STATEMENTS)
B. CHOOSE AN EVALUATION TEAM
C. IDENTIFY KEY INDICATORS
D. IDENTIFY THE INFORMATION SOURCES AVAILABLE AND IDENTIFY TASKS
E. CHOOSE APPROPRIATE TOOLS FOR INFORMATION GATHERING
F. IDENTIFY AVAILABLE RESOURCES (TIME, PEOPLE)
G. DECIDE WHEN THE EVALUATION CAN BE DONE
H. DECIDE HOW TO COMMUNICATE RESULTS (FOR WHOM, WHEN)
I. MAKE RECOMMENDATIONS AND INCORPORATE INTO PROJECT
To facilitate this section, the following example will be used throughout the steps from A to I. This example will always appear in a box.
The village of Kuada* was situated at the edge of a large national forest. The people of Kuada were poor, and the forest was one of their few resources. In the past, they had managed the forest jointly with four other surrounding villages, with rules agreed upon by their leaders. But fifty years ago, the national government took over the management of all forested lands. The government employed forest guards to keep the villagers of Kuada from cutting wood and grazing their animals. But these efforts were, for the most part, ineffectual, and the forest was becoming increasingly degraded. * Kuada is a fictitious village. |
A community forestry project was started by the government. The objective was to co-manage the forest with the villagers. To begin with, ten parcels of seriously degraded forest land were allocated to ten of the one hundred households in Kuada, with the agreement that if this land was managed successfully, Kuada would be given 25 more parcels of land every three years until all interested families obtained land. The project supplied tractors to prepare the ten parcels of land, and Acacia seyal seedlings. The ten families of villagers planted the seedlings, intercropping them with sorghum. In the third year of the project, the community's land was evaluated by the government. The government found that Kuada had not achieved the minimum accepatable stocking rate (400 trees/hectare) and therefore merited only 5 more parcels of land. If Kuada managed to achieve the minimum stocking rate on these 15 parcels of land in three years Kuada, the government stated, would then receive its full allocation. If it did not achieve minimum it would lose all of its parcels. As a result of the government's disappointing findings the village became anxious to formally compare its results with the three other villages which were also part of the project, to improve their management techniques and ultimately their stocking rates. It was decided that a participatory evaluation event would be conducted. |
To establish the reasons for an evaluation, a group meeting of beneficiaries and/or community members and/or others, can make a list of WHY they want or need an evaluation now. It may be useful to review the project objectives and activities at this time. The group can state of what they want to know:
"We need to evaluate our project so that..."
OR
"We need to evaluate our project because..."
OR in some communities, a better response may result from asking:
"What is it we want to know about our project..."
OR
"We want to know..."
If a long list is generated, it may be necessary to prioritize the most important statements.
When the group from Kuada village were asked the specific reasons why they wanted to evaluate the project, they made the following statements: A. "We want to know why our stocking rates are below minimum." B. "We want to know how we can improve our stocking rates, achieve the minimum necessary, and obtain more land". C. "We want to know how, and with what success the other three villages have managed their land, so we can learn haw to have better survival in the future". The facilitator (a project field worker) drew cartoons depicting these statements and also wrote the statements beside the cartoons, on a large piece of white cotton. |
In a meeting, the group can decide whether to have an evaluation team. If they decide to have this team they must then decide who among them will take on this responsibility. If the group is small, this may not be necessary. If the group is large, delegation of responsibility, to those they trust to represent them, may be appropriate.
The villagers from Kuada choose an evaluation team of five persons. They chose: - A woman head of household who had a plot of forest land. A male head of household who had a plot of forest land. - A male farmer who was not involved in the project but was considering participation in the future. - A male primary school teacher who was interested in using the project to teach pupils about communal land management. - The male leader of the community who had negotiated with the project, and chosen the ten households. These people were to gather information that would relate to the statements generated by the group, and report back to the group in one month. |
It will be the evaluation team's responsibility to gather information that responds to the "We want to know" statements that the larger group has listed. The evaluation team should generate a list of their own questions (indicators), which address these statements.
The evaluation team of the village of Kuada came up with four indicators related to the first statement. | |
STATEMENTS |
(Indicators) FOCUS QUESTIONS |
A. We want to know why our stocking rates |
(a) What are current stocking rates? are below minimum. (b) What was the condition of the seedlings when they arrived? (c) How have seedling mortality rates varied over time? (d) Which plots are doing best and why? (e) Have external factors (eg. weather patterns) created unusual conditions? |
After the evaluation team has identified the focus questions (indicators), they must prioritize them to find out which questions are the KEY INDICATORS.
For each of the focus questions, two decisions can be made which should help establish the KEY INDICATORS. One decision is the value of the information, or how important is this question. The other decision is what kind of precision is required, or how carefully does the measuring have to be in order to give a reliable answer.
The value of the information obtained in answering a question can be rated as:
very important
important
not very important
And the amount of precision can be rated as:
very precise
precise
not very precise
From the many focus questions (indicators) that have been generated, the ones to choose are those which are determined most important to answer. Consideration can then be given to the amount of precision that is required.
The evaluation team of the Kuada Village made the following decisions to establish KEY INDICATORS: |
||
Focus Question |
Value |
Precision |
(a) What are current stocking rates? |
very important |
precise |
(b) What was the condition of the seedlings when they arrived? |
important |
precise |
(c) How have mortality rates varied over time? |
important |
precise |
(d) Which plots are doing best and why? |
very important |
not very precise |
(e) Have external factors (eg. weather patterns) created anomalous conditions? |
important |
not very precise |
After much discussion, the evaluation team choose (a), (c), and (d) as their KEY INDICATORS, deciding these that these were crucial to their main evaluation purpose. |
||
At this point it will be useful to find out which questions can be answered with information that is already available, and which questions will necessitate information gathering.
In this step, the evaluation team can look over the KEY INDICATORS and decide what is already known from available SOURCES, whether it is precise enough for evaluation purposes, and what needs to be done.
The Kuada evaluation team identified the available sources of information and the tasks that would be necessary in order 'to answer their KEY INDICATOR questions for the first statement. "We want to know why our stocking rates are below minimum." | ||
KEY INDICATORS |
Sources |
Tasks |
(a) What are the current stocking rates? |
Farmer's Record Books (10) |
gather books, synthesize data |
(b) How have seedling mortality rates varied over time? |
Farmer's Record Books (10) |
gather books, synthesize data |
(c) Which plots are none doing best and why? |
none |
|
The selection of appropriate information gathering tools is very important, as the tools often set the tone for the evaluation and presentation.
The choice of tools may, at the onset, be the responsibility of field staff who are familiar with a range of participatory tools. Some of these tools are listed and described in Section 8, and many of them can be combined, modified, or redesigned to fit the situation.
If the project has been participatory since the beginning, many of the information gathering tools will be familiar. In this case, the community evaluation team can be very involved in the choice of tools.
There are several tools described in Section 8 which can assist with information gathering, analysis and presentation for an evaluation event:
S.W.O.T. Analysis
Popular Drama, Mime, Role-playing
Puppet Theatre
Photographs (slides, drawings)
Participatory Video Evaluation
Tape Recordings
Semi-Structured Interviews
Ranking, Rating and Sorting
The choice of tools will affect the way the information will be analysed. For example, if a drama is to be presented, identification and analysis of socio-economic issues may be most important.
The evaluation team of Kuada Village decided that to gather information on (d) Which plots are doing best and why?" they would use semi-structured interviews. They would interview the 10 participating farmers and also 10 non-participating households. They chose the 10 non-participating households by putting all the names of households (except for the 10 participating farmers) in a basket and drawing 10 names. |
Participatory evaluation events may require the assistance/ employment of people with specific skills, such as interview skills, analytical skills, artistic or dramatic skills.
It is necessary to determine what resources can be drawn upon to assist with the evaluation. Answering the following kinds of questions may help:
"What inside resources do we have?"
"What level of expertise do we need?"
"What outside resources do we need?"
WHO will analyse the information will be determined by the skills of the beneficiaries, the evaluation team or the field staff, but if special skills are required, they can often be developed within the community.
In Kuada, the evaluation team then decided who would be able to do the required tasks. For their first statement and the three corresponding KEY INDICATORS they came up with a list.
KEY INDICATORS |
TASK |
RESOURCE LEVEL REQUIRED |
(a) What are current stockig rates? |
gather books |
*1 person, transportation (no skills) *2 people, 4 hours |
(b) How have seedlig mortality rates varied over time? |
use books |
*2 people, 4 hours (math skills) |
(c) Which plots are doing best and why? |
20 interviews |
*1 person, 10 hours (good interview skills) |
The evaluation team decided that they could distribute these tasks among themselves. The school teacher was chosen to conduct the semi-structured interviews because he was considered to be both objective and trustworthy. |
||
Timing is important. Information must be gathered and analysed within a certain time frame so that the results can reach decision-makers. Seasonal constraints, religious holidays, field staff availability, and off-community labour patterns will be some of the factors to consider in deciding when information gathering activities can take place.
The evaluation team decided that they could work through the following week to do the interviews, since it was a school holiday. A person from the project was asked to visit in two weeks time for one afternoon, to help with the analysis. |
At this stage, the method of "results" presentation is usually decided. New ideas, however, can and may emerge throughout the information gathering and analysis stages.
Information on Communicating Results is presented in Section 7.
Because the whole community was interested in the results of the evaluation event, the evaluation team presented their findings at a community meeting. The results were then further analysed by the community and it was decided that presentation of results to the other three communities would be done using a tape recording of a dialogue of the findings in conjunction with large visual posters of the quantitative results. |
The evaluation event is not complete until the results reach the intended users, decisions are made, and feedback is given and incorporated into the project or activities.
Because the project field worker was assisting with the evaluation events in all four villages surrounding the government forest, they were able to coordinate the results from all four villages and help them to structure the presentation (using the tape recorded dialogue and the visuals made into slides) which would be presented to each of the 4 involved villages, facilitating information exchange. The slide/tape presentation was then used by the project over the next few years as extension materials. Other communities in the country benefitted and were inspired by the lessons learned by the villagers of Kuada. |
Experience shows that participatory evaluation can be a DYNAMIC process, and therefore it may be necessary to monitor the evaluation and change the plan. This can be done by a person, or a team that might ask throughout:
