Community Forestry Field Manual 1

Community Forestry Field Manual 1: GUIDELINES FOR PLANNING, MONITORING AND EVALUATING COOKSTOVE PROGRAMMES
prepared by Stephen Joseph, edited and designed by Carla R.S. Koppell
FAO, Rome 1990


1: Monitoring and evaluation explained

What is monitoring ?

Stove programme monitoring is a process which involves collection and analysis of data to ensure that a programme meets the objectives of the policy makers, donors, programme managers, and most importantly, the needs of the users, stove producers and sellers. It is an essential part of stove programme management.

Stove monitoring programmes can help:

  1. determine whether stove designs are acceptable to all members of a household or whether modifications are needed to make them acceptable;
  2. estimate the number of stoves in use and the frequency of use (this provides a direct measure of acceptance);
  3. determine the stove's performance compared with currently used stoves, thus determining if the new stove is meeting the specifications as set out by the project and established by the users, e.g. relative fuel efficiency, relative cooking time, operating characteristics and lifetime;
  4. determine if targets for adoption and usage are being met;
  5. determine the quantity, quality and cost of stoves produced by factories, artisans, users and extension workers and if production processes could be improved;
  6. determine what other effects the introduction of improved stoves may have;
  7. collect further needs/resource data.

Monitoring can begin in any phase of a stove programme. It is optimal if monitoring is initiated at the beginning of the programme and continues until the programme is completed. But, if a programme that is operational does not already monitor, it is always feasible and adviseable to begin monitoring.

Stove programmes can be divided into a number of distinct phases. In each phase different kinds of useful data can be gathered.

  1. Initial needs assessment
    At the beginning of the programme it is necessary to determine users' needs, understand the existing kitchen management practices, and determine the skills and needs of the people who are currently building stoves, and those who are likely to build improved stoves. It is also important to try to determine feasible testing and distribution strategies.
  2. Design and testing
    Once needs are assessed, existing stoves are modified, or new stoves are designed and tested in some type of a laboratory situation. A small number of stoves can then be given or sold to households to evaluate and test their performance in the field. If these stoves are acceptable, then a more extensive field test phase can be undertaken. If the stoves do not prove acceptable, further development and testing become necessary.
  3. Extension
    During the extended field test phase a number of different strategies for building and distributing the stoves will probably be developed. Training programmes for stove users, producers and extension workers can then be developed and refined. Promotion programmes can begin and staff can be assigned to visit people who have a stove to find out what they think. Further research and development work can help ensure that the performance, ease of construction and acceptability of the stove continually increase and that the cost of the stove decreases. New models may also be developed to meet the needs of different people.
  4. Dissemination
    If this extended field programme is successful, larger dissemination programmes can be undertaken, usually within the framework of a national stove programme. The emphasis of these programmes is large scale construction, distribution and sales of stoves. Production of stoves can take place in households, village artisan workshops, or larger factories. Stoves can be distributed and installed by extension workers, specially trained stove artisans, or groups of trained villagers.

Appendix 1 shows how stove programmes develop. Informed decisions can be made using the monitoring information that is gathered.

What is evaluation?

Evaluation is a process whereby users, producers, project managers, policy makers and funders determine:

  1. whether the objectives of the programme are being met, why/why not; whether the objectives are realistic, i.e. was the problem properly defined in the beginning of the programme and if not, why not;
  2. how the present programme could more effectively meet the needs of households;
  3. the impact of the stove's introduction on specific groups of people or households;
  4. whether or not there is a more cost effective strategy which can achieve the same results.

Much of the data that are used in evaluations are or should be collected during the monitoring programme. Evaluators can collect further information to verify this information.

Evaluation takes place at key points throughout the life of the programme. It can be at the end of the initial field testing phase, at yearly intervals during the extended field test phase and/or at the request of the donor (e.g. midterm reviews).

Criteria and indicators

Evaluation is carried out using criteria and indicators. Criteria are principles or standards against which an objective or technology is to be judged. For example, users might decide that a stove design must meet the following criteria to be acceptable:

Fuel Consumption: 30% less than the existing stove
Cooking Time: 50% less than the existing stove
Level of Emissions: Eliminate all smoke from the kitchen
Aesthetics: Improve the appearance of the kitchen

Some of these criteria can be measured directly, e.g. fuel consumption.

If criteria cannot be measured directly then Indicators are used.

For example, to measure the changes in the appearance of the kitchen, users may decide that the following indicators should be used;

  1. the walls are clean;
  2. extra cupboards have been added and food preparation area extended;
  3. more ornaments and decorations have been added.

To assess whether a stove programme has met its social goals indicators have to be used.

For example, if a programme has the goal of increasing local participation in development programmes, some indicators might be:

  1. greater attendance at community meetings;
  2. an increase in the number of community projects being implemented;
  3. greater number of cooperative actions among farmers.
Criteria and indicators need to be established by users, producers, and stove retailers as well as project managers and donors.

Examine the different criteria and indicators that different groups of participants suggest. It can help in planning, designing, monitoring and evaluating. Appendix 2 provides examples of possible indicators.

Who undertakes evaluation?

Evaluation can be undertaken by programme staff, outside consultants or personnel from the donor agency, and users and producers of the stoves. External evaluators can take a fresh look at the programme, and since they are not closely involved with its success or failure they can be more objective in their assessment. However, external evaluators may misinterpret the information they collect from the programme staff, producers and users, especially if the local staff find the external person threatening. Evaluation is most effective when a team comprised of external agents, programme staff, users and producers work together.


BACK TOP TABLE OF CONTENTS BOTTOM NEXT

2: Planning and managing stove programme m & e: a participatory approach

Effective management of the stove programme depends on availability of specific data at specific times in the programme's implementation. To ensure the timely flow of information, careful planning and management is necessary.

To plan a monitoring programme there must be:

To properly manage the monitoring and evaluation of a programme, there needs to be:

Managing stove programmes

Stove programmes have been most successful when the users and stove producers have been involved in the stove's selection and/or design, laboratory and field testing. Successful widespread dissemination of the technology is more likely to occur if users, extension workers and producers develop and test extension programmes together. It is important to closely involve entrepreneurs, artisans and small shop owners in the development and testing of marketing strategies when programmes use a commercial approach to achieve large scale dissemination. Effective participation is crucial to good management of a stove programme.

A guide to management

Over the last 5 years a number of programmes have developed methods for participatory management of stove programmes. Their experience has been synthesized to provide the following field tested approaches.

Modify this programme to suit local needs and conditions. For example, the methods used in urban areas are usually different from those used in rural areas.
STEP 1

At the beginning of the programme, establish a team including a project manager, a field worker, a stove designer, a respected artisan and a group of people who want to be involved in the stove programme in their area. The local stove committee, LSC, may, for example, be drawn from a local women's, social or religious organization, chosen by the village community to help guide the programme. In urban areas the LSC may include a stove factory owner, a distributor and a retailer as well as representatives from other local groups and users. The LSC's activities can be closely coordinated with the work of other groups.

STEP 2

The LSC carefully defines its role, how it is to operate and when it is to meet. It also decides how it is to communicate with the community. It is extremely important that the programme staff do not dominate. Their role should be to listen.

Because most people's time is extremely scarce and valuable it may be necessary to consider some type of payment for the time that is taken and the work that is done. This may be in the form of cash, food or provision of new stoves at no cost. A cash value assessment of each community participant's work should be used to determine appropriate payment. By assessing work time on an individual basis, the potential for conflict over "appropriate" payment is reduced. Care should be taken that the reimbursement does not cause resentment among other members of the community.

STEP 3

The potential users should be helped to carefully define their needs and develop a set of design criteria with the stove researchers. There are some basic questions that can help the participants determine their needs and objectives (see Appendix 3). In rural areas, villagers should also establish their own set of programme objectives, and the necessary criteria and indicators against which these objectives should be monitored and evaluated. In some urban areas, where there is a strong community development activity, local people should be involved in programme design. Appendix 2 provides information that can help guide the team so that it gathers the information that is necessary for producing a set of criteria and relevant indicators. This information is used at different times throughout the life of a programme. The information needs of different phases are outlined in Appendix 4. Also included in Appendix 2 is a checklist that is an example of an Initial Needs Assessment Survey. It can be adapted for use.

STEP 4

The villagers and urban dwellers should also help designers catalogue the existing stoves, measuring their performance, in both the laboratory and the households.

STEP 5

Members of the local stove committee, designers and extension workers should monitor the acceptability of the early prototype stoves. Regular meetings of these people and project management staff can help assure that necessary design changes are quickly implemented. Make all necessary modifications with the participation of the stove group. The community members, with the assistance of the designers, are often the best people to lead the modification process.

STEP 6

In the extended field testing phase the local extension agent or person in charge of marketing should report to and work closely with the local stove committee. If possible the LSC should have a major say in the appointment of the local extension agent. Experience from other programmes indicates that community meetings need to be arranged by the Committee at certain key times to resolve problems and/or retain the interest and enthusiasm of the rest of the community. It is extremely important that senior management attend these meetings.

STEP 7

The type of participation will probably change once a programme enters the dissemination phase. Programmes are most effective if a national stove programme is established and all institutions collaborate to monitor the national programme. It helps if all the institutions use the same or similar methods to collect and analyse data.

In urban areas, distributor and producer monitoring can be carried out by the programme staff or professional marketing consultants. Local stove groups, if they are interested, can have a major role reporting the effectiveness of the promotion campaign, the quality of stoves being sold in the market and the acceptability of the new stove. The individual consumer, however, should be encouraged to report on the acceptability of the new stove, possibly by filling out a warranty card or answering requests through the media.

In rural areas the form participation will take depends upon the types of stoves being promoted and the dissemination strategy being used. For the dissemination of mud and mud-lined ceramic stoves, monitoring can be carried out by village groups and extension officers or it can be assigned to a particular individual who receives special training. For portable ceramic stoves sold in the local market, monitoring can be done by district industrial development officers or a specially trained artisan.

Structures for managing monitoring and evaluation

In very small programmes (where the target is approximately 2000 stoves), the monitoring programme the LSC has outlined can be overseen by the programme manager. This person may have an assistant who is responsible for working with the community members collecting and collating the data from the field. During the initial field test phase the manager may have to spend a lot of time working with users and/or stove builders. Once dissemination of the stove commences the manager can spend less time with users and more time overseeing monitoring. The manager schedules regular meetings with the extension workers, users and producers and design team to analyse data and develop solutions to problems that have arisen. Problems and successes should always be discussed in LSC meetings. The LSC should work together to make decisions and choose solutions. The manager must insure that consultants who are sent to evaluate the programme meet and work with the LSC.

For larger rural based programmes a monitoring and evaluation unit may need to be established. The manager of this unit often reports directly to the programme manager and the LSC. The M & E manager:

  1. supervises and monitors the work of the field staff, consultants and other participating organisations;
  2. provides regular feedback to the team manager, the LSC and the design team;
  3. organizes and chairs the LSC and field staffs' regular review meetings;
  4. organizes and participates in field staff training;
  5. scrutinizes and/or devises the initial survey. Supervises checks of collected data;
  6. helps the LSC and other community members prepare reports and ensure that they are presented on time.

The M & E unit will probably need to work closely with both the research and development, and the extension departments. The M & E unit will usually have an input into training programmes other departments run. Regular meetings will need to take place to quickly solve stove design and extension strategy problems.

M & E units often have one or two additional staff who should, if possible, have both technical and social science training. They can assist/ supervise the daily activities of other enumerators. Much of the detailed participant observation and measurement can be carried out by these trained social scientist/technicians. They can organize the regular review meetings with, and training of the field staff and the locals. They can assist, collating, sorting and analyzing the data and helping locals draft reports.

It is always possible for more local people to become involved. Different tasks can be assigned to small groups who, with the help of trained staff, can refine and implement the M&E process that has been outlined by the LSC.

In urban based programmes monitoring and evaluation is an integral part of a marketing strategy. Thus the manager of the marketing programme can oversee urban M & E.

A simple planning framework for monitoring

STEP 1: Determine the objectives and principal monitoring and evaluation activities for each phase of the programme.
There are five major steps in monitoring and evaluation. The team can work together to develop a set of programme goals for:

  1. stove performance (and criteria to be used to judge stove acceptability)
  2. usage rates
  3. stove efficiency and production quality
  4. marketing and/or extension strategy effectiveness
  5. rates of adoption and impact

Small programmes may not want to do all of these, however, it is strongly advised that b, c and d be undertaken.

STEP 2: Determine the minimum information requirements for each phase.
Appendices 3 and 4 provides information on minimum information requirements.

STEP 3: Determine the method of collection, storage and processing for each phase to determine the minimum information required.
Section 5 provides information on personnel needs and training for data collection. See Section 3 for detailed information on data collection techniques.

STEP 4: Determine what activities must be undertaken and when they should be undertaken.
Undertaking activities in the following order can help insure the effectiveness of the monitoring:

  1. train staff and community members;
  2. develop a method for monitoring with users;
  3. select households to be monitored (sampling);
  4. develop and test the information collection and the storage methods;
  5. purchase/organize the equipment and transport that are needed to test the stoves and complete the surveys;
  6. undertake and monitor field work;
  7. process and analyse information;
  8. implement review meetings;
  9. write and communicate reports to all relevant parties.

Graphics often clarify responsibilities, sequence, and timing of activities and relationships. Charts 3.1, 3.2 and 3.2 illustrate how different types of tables and charts can be useful planning tools.


YEAR 1
OPERATION J F M A M J J A S O N D
1.Initial staff training --- --









2. Stove installation
- ---








3. Start monitoring

--- --- --- --- --- --- --- --- ---
informal interview

--- -







select sample

- -







develop questionnaire

-- -







check stoves


-- --- -





administer questionnaire


-- --- -





analyze results




-- --- ---



present & report results







---


4. Design stove alterations








--- --
meet team








--- --
design changes









---
5. Distribute redesign










---

Fig. 3.1 A detailed bar chart for planning time allocation

Main components of m&e work programme Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
1.Training ----- --









2.Installation
-----









3.Monitoring

----- ----- ----- ----- ----- ----- ----- ----- ----- ---
4.Evaluation




----- ----- -----



5.Redesign & distrib.







----- -----

Fig 3.2 A simple bar chart for defining tasks

Activity Time needed Human resources Notes
Training 1.5 months All: producers, users, promoters, designers, etc. Phased: 2 weeks each group
Installation 1 month Installers, Designers Designers and ISC manage and oversee
Monitoring ongoing from installation Enumerators
Evaluation 3 months Designers, Staff Begin after collection of monitoring data.
Redesign 1.5 months ISC, Designers Integrate information from the m&e.
Distribution 1 month Installers

Fig 3.3 Time and resource planning chart

A bar chart is the easiest method to display this information. Some activities must be carried out before other activities. Bar charts can be detailed to map out specific time allocation (Figure 3.1). They can also be very simple to show tasks that need to take place (Figure 3.2). A time planning chart (Figure 3.3)shows use of staff resources as it relates to the project's M & E scheduling.

STEP 5: Determine the amount of money, equipment and the personnel required to collect, process, analyse and report the information in each phase.
Once all the activities are specified, determine who is to carry them out, what equipment will be needed and the costs that are involved. Ensure that those people who are assigned particular monitoring tasks do not also have other tasks assigned for the same period. Figure 3.1 illustrates how this information can be laid out. The minimum resources requirements are:

  1. A minimum 1% to 4% of total budget; in the initial field testing phase monitoring can require 30%.
  2. 30%-50% of designers' time during the initial three months of field tests.
  3. 30%-50% of extension workers' time and 20%-40% of designers' time during the remainder of the field test phase. 10%-20% of extension workers' time during the expansion phase.
  4. 5% of programme managers' time (where she/he is not the designer).
  5. Balances, rulers, watches, log books and bicycles.
  6. At least one week per year devoted to teaching monitoring techniques.

STEP 6: Determine if the available resources are sufficient to collect and analyse the information.
Having calculated the number of people, days and the money required to carry out the monitoring and evaluation programme, determine if sufficient funds are available or could be made available. If it is apparent that the monitoring programme is too ambitious the type and quantity of information will have to be reduced or cheaper methods for collecting the same type and amount of information will have to be developed.

STEP 7: Develop an organisational structure that involves users and producers.

Communication and reporting

The importance of communication cannot be overemphasized. It can be helpful to hold review meetings with the headquarters and field staffs at least once every three months. The head of the M & E unit and the programme manager can meet at least once a week and make regular field visits together (if possible once every two months). Where possible the two should also attend review meetings with donors, civil engineers and/or politicians. If a consultant is permanently attached to the M & E unit, it can help if he/she regularly meets with the manager. Community groups, stove designers and other personnel involved in monitoring, can also meet.

In larger stove programmes report writing is a very useful form of communication both for internal use and for informing donors and the wider public. It is recommended that reports for internal use be short (under five pages). They should state the objectives of the particular monitoring exercise, the work undertaken, the results, and the short and long term action that is required. Guidelines to follow when writing a report are:

1)KEEP IT SHORT by emphasising key points and using tables to summarize information.
2)KEEP IT CLEAR by planning the report carefully, avoiding technical jargon, using short sentences and illustrating points graphically.
3)EDIT CAREFULLY Leave a day or two and let someone else look at the report and comment before doing a final edit.
4)SUBMIT ON TIME This not only assures there is quick action to resolve problems, it also builds confidence in the M & E unit.

At the end of each programme phase a longer report can be issued for general circulation. The following could be included:

  1. general outline of project objectives
  2. background
  3. details of monitoring and evaluation methodology
  4. results of monitoring
  5. conclusions
  6. resulting modifications to the stove's design and the extension programme that are needed

BACK TOP TABLE OF CONTENTS BOTTOM NEXT

3: A simple participatory approach to data collection, analysis and use

The method

This monitoring system is based on the experiences of programmes in both Asia and Africa and can be used in small and large programmes. It involves data collection by local people working with stove programme personnel, and immediate data analysis of the data using a question and answer format. There are other approaches to data analysis that can be used with this monitoring and evaluation system. They are described later in Section 3. The participatory approach has certain advantages:

  1. is relatively inexpensive,
  2. provides feedback to designers and programme managers quickly,
  3. can provide more reliable information;
  4. increases the skills and confidence of local community members who are involved in the monitoring,
  5. helps transfer much of the responsibility for implementation to the local community.

It also has some potential disadvantages:

  1. Any bias within the local group of people will be reflected in the sample that is chosen, and in the data that is collected and analysed.
  2. Since there is an emphasis on verbal communication and reporting, project staff will have to spend time recording and then synthesizing the discussions.
  3. It will be more difficult for outsiders to evaluate the data because most of the context is gained during the discussion and large amounts of data are collected using unconventional techniques which do not generate statistics.

Some programmes may wish to use a combination of the described participatory and more conventional techniques. Throughout the text relevant complementary Appendices are also listed.

Adapt these techniques to meet the programme's needs. The following sections guide development of a programme-specific participatory monitoring system.

Collection of baseline data

STEP 1

At the beginning of the programme, work with members of the community, local artisans and entrepreneurs, extension workers, government officials, and the local stove or development groups to gain a profile of :

  1. the housing (and in particular the kitchens),
  2. the fuel and food situation,
  3. the local artisans, resource availability, small business and distribution system.
  4. other relevant information to be used when designing stove, extension or marketing strategy. (This may include information on relative wealth of the different individuals, history of other appropriate technology programmes, and institutional profile etc.)

For a general list of the types of baseline information that will be needed for a complete evaluation in the later Initial Field Test Phase see Appendix 4.

STEP 2

Information can be gathered and verified if the stove committee develops questions which can elicit important information.

It is extremely important for the stove committee to develop a set of questions and then seek out answers.

Have a member of the group that is familiar with the monitoring material serve as facilitator. The facilitator can ask a series of questions and then help the community members or the local stove group find answers. One person, who has already reviewed the information collected from the monitoring, acts as facilitator ( such as the head of the M & E unit from the stove programme). For example, the stove group finishes a survey on the community needs and is examining the data. The facilitator asks a series of question such as:

"How will the size and layout of the kitchens affect the design of the stove"

He or she then asks for possible answers from the group. These might be:

"I have noted that 8 out of 10 people that I visited, have very small kitchens in which the stove is placed in the corner of one room. Often the stove is against an internal wall so that heat can be provided to the rest of the house. I feel that the new stove should not be bigger than the existing open fire and that, if a chimney is to be used, it should be taken through some of the other rooms."

The facilitator can ask for other monitors' comments on kitchen size and its effect on stove design. The group builds up a picture and draws some conclusions on the shape of the stove. For example:

"We have now determined that approximately 80% of the people in our area will want a stove that has dimensions that are no more than 40 cm wide and deep. The other group of people, with bigger families and bigger kitchens want a much bigger stove."

After drawing this conclusion further analysis of the data can take place until a full set of design criteria are drawn up. This can then be passed on to the stove designers.

The following give an indication of the range of questions that could be used and their possible answers. There are other methods that can be used for participatory data collection and analysis. These are described in Section 4.

The kitchen

Kitchens vary from country to country and within countries.

They can be a part of the main house or a separate building. In very hot climates, preparation and cooking of food may be done outside during part of the year. Within the house the kitchen may be situated on the ground floor, the middle floor or higher floors.

Some kitchens are small, have very low ceilings and little ventilation. Other kitchens are very large and are well ventilated. Roofs are sometimes made from thatch and sometimes from tile or iron.

Some kitchens are very clean and have special storage places. Kitchens are organized in many different ways. In some, the stove is placed in the middle so that people can eat and socialize around the fire. Some people have special places for worship in the kitchen and some must put their stove in these sacred places.

The following are examples of questions related to the kitchen. Try to predict the answers; it can help design the stove and the strategy for field testing. Sample answers from the Division of Community Forestry Programme in Nepal will be given to guide the reader (Joseph 1982).

Question 1:
How does the position of the kitchen affect the choice and the method of installation for the stove?

Possible answers:

  1. If people change the site of the kitchen during the year, a fixed stove will only be used part of the time. It may be necessary to either introduce a portable stove or put in 2 fixed stoves.
  2. It is very important for users to decide the stove's primary function and the room in which the stove should be placed.
  3. Chimneys are much more difficult to install in multistorey houses that have the kitchen on the bottom floor.

Question 2:
How does the size of the kitchen affect the choice and installation of the stove?

Possible answers:

  1. Large stoves do not easily fit into small kitchens. If the stove is too big it can leave too little room for food preparation or socializing around the stove.
  2. chimney is difficult to install if the kitchen is situated on the bottom floor of a 2 or 3 storey house and its height is less than 1.5 meters. A series of tight bends must be placed in the chimney and this reduces the draft and makes cleaning difficult. If the user wants smoke to be removed from the kitchen it may be necessary to build some type of chimney hood.

Question 3:
How does the degree of ventilation and the level of lighting affect the method of installation that should be used for the stove?

Possible answers:

  1. The new stove should have lower levels of carbon monoxide emission than the existing stoves if a chimney is not to be used.
  2. If people cannot afford to use more kerosene, any new stove must provide the same amount of light as the old stove (especially in kitchens with low levels of natural light). It may be necessary to open up the front of the firebox.

Question 4:
How does the condition of the kitchen affect the choice and use of a stove?

Possible answers:

    People who take time to maintain their kitchen will probably want a stove that looks attractive, can be easily cleaned and does not blacken pots. It is probable that less effort will be required to train these people to maintain the new stove.

Question 5:
How can the type of roof on the house affect the choice and installation of a new stove?

Possible answers:

  1. For some people it is important to keep the thatch dry and free of insects by allowing the smoke from the fire to come into contact with it. In this case it may be necessary to place the chimney under the thatch or to use a non-chimney stove.
  2. If people want the smoke removed from their thatched roof house it will be necessary to ensure that a barrier is placed between the thatch and the chimney.

The stoves

In most countries a range of stoves/fireplaces are used to cook food, prepare beverages and sometimes process animal feed. Households may have more than one stove. Different fuels may be used and different cooking operations may be performed on different stoves. Some stoves are fixed, others are portable. Some stoves must carry large round bottom pots that are vigorously stirred and others are used with light weight small flat bottomed aluminium pots.

Most stoves are very cheap to build or purchase. In urban areas they are usually made from metal and in rural areas from mud, ceramic or stone. Most ceramic and metal stoves are made by artisans, most mud and stone stoves are made by the users. In some societies stoves can only be made on special days and in others communities they must be blessed by the local priest. Some women prefer to squat when cooking, others prefer to stand. Some women are very careful tending the fire, others pay little attention to it (especially if they have demanding children with them).

Most simple stoves (e.g. the 3 stone fire) can provide both light and heat, use a range of fuels and accommodate a range of pot sizes. They are long lasting, require very little maintenance and if operated carefully have a relatively high efficiency. However, it is often difficult to burn wet wood and residues, protect the fire from wind (which considerably reduces the efficiency), prevent small children from burning themselves and prevent large volumes of smoke from being emitted.

Question 1:
What are the implications for the design and choice of different new stoves, given the range of stoves used by different ethnic groups?

Possible answers:

  1. In many areas it will be necessary to introduce different models to different groups. Sometimes two or more stoves are needed to fulfill all a group's cooking needs.
  2. Each model may require a range of options such as 1,2, or 3 pot holes. A stove may need to be designed so that it can be used with or without a chimney.

Question 2:
How will diversity in the types of stoves (and the level of their social and cultural significance) effect the introduction of improved stoves?

Possible answers:

  1. It is essential that extension workers spend considerable effort discussing all the functions of the old stoves. They must consider with the user what functions the new stove can perform, helping the family decide if they will need to keep their existing stove for some purposes.
  2. People may be more interested in buying a new stove if its longevity and cost are similar to their current stove.
  3. It is much more difficult to train people to maintain and repair a new stove if they haven't had to do this previously.
  4. If women normally make or purchase the stove/fireplace, involve them in new stove selection and installation.
  5. If men must give permission to purchase a new stove they must be involved in the extension effort or be targeted in the marketing programme.

Pots

In most societies a range of pots are used to cook food. Pots are made from clay, cast iron or aluminium and have either round or square bottoms with either flat or curved walls. Most families will have a range of different size pots and often these will be used for cooking different meals.

Question 1:
What are the implications of different pot types and sizes for the design/choice of the stove?

Possible answers:

  1. It is most important that the stove be able to accommodate all of the most frequently used pots.
  2. Moulding the pot seat to the shape of the pot can provide extra stability when cooking.
  3. For a two-pot stove, provide sufficient space between the first and the second pot seat to accommodate the 2 largest pots when they are used together.
  4. If big pots are used most of the time, the pot seat diameter (of the first pot) should be smaller than the smallest of the these big pots. If small pots are to be used occasionally a specially designed ring, to reduce the pot seat size, should be provided with the stove.

Question 2:
How will different pot types and sizes affect installation of mud and ceramic-and-mud stoves?

Possible answers:

  1. It is important that stove promoters, builders and installers get women to bring out all their pots and identify their uses. Determine the distance between pot seats and chimney, the inner and outer diameter of the pot seats and the camber that will enable most of the pots to sit safely on the stove.
  2. Consider and discuss the possibility of building other stoves to accommodate larger pots.

Fuel

In most rural areas people use a range of biomass fuels. The type of fuel that is used depends on many factors including, price, people's income and occupation, availability of different fuels at different times of the year, and location of different types of fuel.Some people store wood fuel, others collect it every week; some people purchase their fuel either in bulk or in very small quantities.

Some species of wood are easy to burn while others are difficult. Some burn with a long flame, others with a short flame (especially charcoal). Some people chop wood into small sizes, other people use very large pieces.

Question 1:
How will different types of fuel affect the stove's design?

Possible answers:

  1. It is extremely difficult to design a stove that burns all types of fuel efficiently. Residues and wet fuel require more air to burn properly and thus a stove should have secondary air holes and a facility for drying fuel.
  2. Some softwoods and many residues produce more tar and soot than hardwoods. Thus chimneys must be cleaned more often if chimney fires and blockages are to be prevented.
  3. Some fuels produce a lot of ash and char which can block passageways. Passageways need to be carefully designed to prevent blockage.
  4. Little heat will get into the second pot hole when wood that burns with a short flame is used. This will affect the distribution of heat and the speed of cooking. It may be advisable to develop a stove in which both pot seats are placed over the fire.
  5. A large door is required if large pieces of wood are used.

Question 2:
How does the type of fuel and the purpose of the stove affect marketing/extension strategy ?

Possible answers:

  1. If households use low quality fuel, much more training is required.
  2. People who pay for fuel are more likely to realize the benefits of the new stove quickly, they will therefore respond to a promotion campaign faster.
  3. Those people who have the greatest difficulty obtaining fuel are often the people who do not have access to extension services. It may be necessary to identify these groups and develop some affirmative action programmes. However, it should be realized that these people are often the least likely to risk investing in an innovation.

Food

The amount, type and number of times food is cooked each day varies between families, villages and regions in any one country. In some urban areas people find it cheaper to eat in restaurants than to pay for food and fuel. If people travel long distances to work, they will often cook food the night before, heat it up for breakfast, and take the leftovers for lunch. During harvest time cooking is often done once a day.

In some places rice or other grains are staples, in others root crops such as potatoes; in some places no meat is eaten, in others it is a major part of the diet. Some food is cooked quickly, others take many hours of slow boiling. Some people want two main dishes to be prepared at the same time, others will cook dishes sequentially. Some people use their stove for the preparation of food and drink for sale.

Question 1:
How will the range of meals that are cooked in any one household affect the design of the stove?

Possible answers:

  1. For people who prepare only one dish or cook for less than 30 minutes, it is important to design a stove that heats quickly.
  2. When food is to be stirred vigorously it is necessary to ensure that the pot is well supported and that the stove is stable.
  3. When two dishes need to finish cooking at the same time it is necessary to partition the heat equally between the first and second pot .

Similar profiles and questions can be developed for manufacturers, distributors and retailers. Other socioeconomic data can also be examined and the relevant questions formulated and answered. Once these questions have been answered the stoves' design is likely to meet people's needs and fulfill the criteria that have been established. The programme can now field test the stoves.

The initial field test phase

STEP 3

The project personnel and the local stove or community group will now have to decide those who will test and evaluate the first series of stoves. Involve a representative sample of households from all the different groups in the area. Place the stoves in the households and if possible, visit within one week of installation and then again within 3 months. Members of the local stove group (if available) should accompany either the local people employed to do the extension work and/or the outside project staff.

STEP 4

In-depth discussion should take place with the stove users during the visits. These discussions should be recorded in full, even if they seem unrelated to stove usage. Designers will also need to work with local people to collect information on the performance of the stove. Local people can increase project staff and stove committee understanding of the social organisation and interaction of the community. Hence, they can provide insights into the results of the data analysis. For example, a woman stopped using the stove, but when asked why, would not give an answer. However, she did insist that the stove was acceptable. During discussion she had mentioned that her stepmother had just come to live in the house. One of the local people involved in the monitoring felt that the stepmother was probably the reason why the woman had stopped using the stove. Later the social scientist asked if the arrival of this person had influenced her decision to use the stove. She said yes and explained that she felt too embarassed to answer the question in front of other local people. After collecting data from all the houses, the answers to the questions should be reviewed and analysed by the stove designers, programme managers, local extension workers and as many members of the community as possible. Section 3 provides several methods for group data analysis.

The following set of questions, answers and actions are given as a guide to develop an interview checklist and then to analyse the results. It will be necessary for each programme to make modifications and additions.

Question 1
Why are people not using the stove ?

Possible answers Possible actions
it cooks slowly
  • increase the size of the chimney, secondary air, wood entrance
  • increase the size of the combustion chamber and/or pot holes
  • clean the stove
  • increase the number of pot holes
it uses too much fuel
  • decrease output of the fire
  • increase the size of the baffle
  • reduce firebox height
  • reduce the wood entrance size
it produces too much smoke and is difficult to ignite
  • clean out the chimney and passage ways and retrain users
  • increase the chimney height
  • increase the size of the firebox
  • add more secondary air
not enough light and heat
  • increase the size of the front opening
cracking or breaking of walls
  • change mud or clay mix
  • change firing temperature of clay
  • improve training of artisans
  • improve training of users
is uncomfortable to use
  • change the height
  • increase the level of insulation
built in wrong position, or time of year, death has occurred, or women/men were not consulted
  • retrain extension worker to be more sensitive to cultural factors and gender issues
  • talk to users about rebuilding the stove at the right time, in the right position

It should be noted that some of these actions may conflict. For example to increase the speed of cooking you may find that the fuel consumption is increased. The users need to work with the stove committee and set an order of priorities.

Question 2
What do the users best like about the stove ? What are their indicators for success ?

Possible answers:

  1. The stove saves time;
  2. reduces fuel consumption and therefore also reduces either the time that has to be used collecting fuel, or the money that needs to be spent on its purchase;
  3. reduces level of emissions;
  4. reduces the time needed to clean the kitchen and pots.

Possible actions

  1. Include these points in the training of extension workers.
  2. Highlight the main features that users mention in the promotional campaign.
  3. Help the community decide if and how it might like to share its experiences with others.

Question 3
What changes to the users' kitchen management practices have been noted as a result of the new stove's introduction ?

Possible answers:

Women have made many different changes to their kitchen management practices. Some households have built a new, more spacious kitchen as a direct result of the stove's introduction. Others have built more storage and preparation space around the new stove. Some have started boiling more water for drinking and washing (as they have more time and fuel). In Guatemala, women have been able to cook their beans in the retained heat of the large mud stove overnight, instead of early in the morning. In some African countries women have been able to spend more time caring for their children while the food is cooking (the new stove requiring much less attention than the open fire).

Possible actions

The improvements to the kitchen that have been made can be publicized. Incorporate the methods women have used to improve the kitchen or cooking techniques in the training programmes. Organisations that are involved in women's development and housing improvement programmes can be informed of these changes; they may introduce them into their programmes.

Question 4
What improvements do people want?

The answers and actions will basically be the same as those presented in Question 1, however additional possibilities could be:

Possible answers Possible actions
improve looks
  • paint it
  • put a decoration or stamp on it
  • provide curved surfaces
make lighter
  • change the materials
  • reduce wall thickness
reduce outer wall temperature
  • increase wall thickness
  • put in higher quality insulation

The extended field test phase

Once an acceptable stove design has been developed the programme can develop either an extension strategy for stoves that are to be distributed through a government or non-government agency, or a marketing strategy for programmes that are to use small industry and business to produce, distribute and sell the stoves.

STEP 5

The technique for collecting information in this phase will be different for stove programmes based in rural areas that use an extension strategy, and those in urban areas that use a commercialisation strategy. For rural programmes, monitoring staff, stove producers, extension workers, stove installers and members of each village stove committee will have to agree on a series of questions that need to be asked and then, if possible, participate actively in the monitoring. Analysis of the results can take place in each village using the techniques described in STEP 2 and/or later in this Section.

In urban programmes much of the monitoring can often be undertaken by the people involved in marketing (although some programmes have a separate monitoring unit). Monitoring forms and questionnaires can be developed in collaboration with the stove committee. It is unlikely that most of the members will have the resources to participate in the actual collection of data. The committee should help the personnel involved in monitoring select the sample, develop strategies and techniques for information collection and analysis and formulate a plan of action to improve implementation of the programme and the design of the stove.

In the dissemination phase other questions regarding reasons for adoption/non-adoption need to be answered. For a list of information projects could gather during the extended field test phase see Appendix 4.

Question 1
Why are people not adopting the stove? What can be done to increase adoption?

In urban areas:

Possible answers Possible actions
have not heard of it
  • increase promotion
can't find it
  • increase distribution outlets
  • increase level of production
  • increase training of artisans
  • improve profits of producers to ensure sufficient incentive
unconvinced of time/fuel savings
  • organize demonstrations/increase promotions
don't like look
  • change external appearance/hire industrial designer
uncertain of lifetime
  • provide warranty
can't accommodate all pots or can't cook all types of food
  • produce several models
husband not convinced of benefits
  • reorient promotion campaign

In rural areas:

In rural programmes, especially those using a commercial strategy, the same answers will be heard from users and the types of action can be taken. However, in programmes that use an extension strategy, there can be other reasons why the stove is not adopted; these are specific to both the type of stove and the extension strategy being used. For example:

Possible answers Possible actions
extension agent has not visited
  • examine their schedule
extension agent has not properly considered users' needs
  • retrain extension worker
extension agent is seen by users as poor on follow-up
  • improve field management
requires new kitchen before will adopt stove
  • link project with other housing community development efforts
a death has occurred in the family
  • revisit after suitable time
waiting for a subsidized stove
  • rethink subsidy programme

Question 2
Why are people adopting the stove?

Possible answers and actions:

There are many reasons why people adopt a stove. Most frequently it is because they remove smoke, reduce cooking and/or cleaning time, and fuel consumption (which results in a reduction in the time spent collecting fuel or in the amount of money spent each month). Stoves may also be adopted because they improve health, reduce injury to children, increase status or self-worth (a woman may exclaim that she feels really good after buying a new stove), because it is part of a social group's development programme or because of peer pressure.

Managers must fully understand the different reasons why different groups of people are adopting the stoves and reorient their promotion programme to increase the rate of adoption. It is extremely important to try to predict the increase in the adoption rate for a given increase in promotion since the necessary resources/personnel must be available to meet the demand as it increases.

Question 3
Is the stove programme meeting its operational objectives ? If not, what additional resources are needed?

Possible answers and actions:

In any programme there will be several operational objectives that may be social, technical, economic and/or institutional.

Many of these objectives are related to targets which could include:

  1. the number of stoves produced, distributed and either sold or installed,
  2. the number of users, extension workers, artisans, entrepreneurs, distributors, retailers and programme staff who receive training,
  3. the usage rate for each year,
  4. the overall fuel savings and/or the reduction in cooking or wood collection time and emission levels,
  5. the profitability of the stove enterprises and the return on investment,
  6. the average cost to disseminate one stove (at the end of a certain period of time),
  7. the number of local groups formed to handle production, distribution and promotion,
  8. the number of other activities that should result from the stove development effort.

Most of the reasons why a programme reaches or does not reach its targets have been given in the preceding sections. For example, consider Objective 1 above). The reasons why the programme may have failed to reach its production and distribution targets and the actions that are needed to remedy this could be:

Possible answers Possible actions
insufficient promotion
  • increase quantity and quality
insufficient training
  • increase training
poor follow-up
  • improve supervision and increase communication with locals
insufficient incentive for producers / retailers / distributors
  • increase price
  • lower production and distribution costs
  • change design/production method
poor production management
  • increase training

However, targets may also not be reached because of inaccurate project appraisal and poor planning. If unrealistic targets and objectives have been set, use these as a lesson when designing the dissemination phase of the programme.

Question 4
Is the new stove having an impact ? Which impacts are beneficial; which are detrimental?

Criteria and indicators for assessing the stove programme's impact need to be developed by local monitoring groups along with other producers and distributors. Once criteria have been established, the answers can usually be ascertained from the data that were required to answer the other questions. For example if the group wants to find answers to the question "What is the impact on producers?"

They can find the necessary information from Questions 3 and 1. The answers from this impact evaluation can assist development of the strategy for the next dissemination.

STEP 6

At the end of the extended field phase an evaluation is usually carried out either by the project itself or by the donor. Since the programme has been monitored by the users, producers and programme staff, an internal evaluation could involve the following (this technique has been used by an NGO in India):

  1. The local stove groups, and the programme staff separately review the material that has been collected.
  2. A report is made by each group. For illiterate people a tape recorder or a scribe can be used.
  3. The reports are circulated at least a week before a meeting is held.
  4. These reports are then presented at the meeting.
  5. An unstructured discussion ensues.
  6. This is followed by a structured discussion.
  7. The discussion is then written up (or summarized on audio/visual tape) and presented to all parties for their comment.
  8. Once all parties approve the report, it is published. For illiterate villages, a video/audio tape is produced.

Discussion (managed similarily to intermediate stage evaluation discussions) can answer these questions:

  1. Are the objectives of the programme being met, why/why not ? Were the objectives realistic; i.e. was the problem properly defined in the beginning of the programme, if not why not?
  2. How could the present programme more effectively meet the needs of households?
  3. What is the impact of the introduction of a stove on a specific individual or household?
  4. Is there a more cost effective strategy that can achieve the same results?
  5. Should a dissemination phase be undertaken and if so how?

The community and donor may bring in an outside consultant to help in the evaluation. If so, it is strongly recommended that this person be versed in participatory techniques. The outside consultant needs to first become familiar with the project and meet some of the local people that were involved in its implementation. The consultant can review all the previous reports and write a summary. This would then be presented at the evaluation meeting for comment and criticism. The consultant can finalize the statement and it can be presented either individually or as a part of the final report.

STEP 7

If the project is expanded, a monitoring system will have to be established. If the dissemination is to be conducted by other small organisations then the monitoring system that was developed in the previous phase can be used. The coordinating organisation can train the other organisations in the techniques that have now been fully developed and tested. Training can be adapted to suit the needs of different organisations. However, it is recommended that a participatory approach be maintained.

The information needed to monitor a dissemination programme is the same as for the expanded field test phase, however the priorities placed on collection of different types of information will change. Thus in the dissemination phase the programme management will be more concerned with collecting information on:

  1. the number of stoves being produced and the quality of production and
  2. the effectiveness of the extension/marketing programme.

Thus the programme will collect more information on categories a to l in b) of Appendix 4.

In larger urban and very large rural programmes that use a commercial strategy it may be extremely difficult to use a participatory approach in all areas. Field experience in both Africa and Asia suggest that information on sales, production quality, output, and the number of new outlets being established are more effectively collected and analysed by a full time project person. There appears to be little need to set up a number of stove committees. However, experience in Sri Lanka suggests that regular meetings between producers, distributors and stove staff should be organized. During these meetings the monitoring data can be reviewed, and further data can be collected during discussion. At the end of the meeting a set of proposed actions for improving the programme can be drafted by all the participants. In both the urban and rural settings monitoring can gain information which answers specific questions:

  1. Does the initial rate of adoption meet the targets in the project plan? Who is adopting the stove and are they the intended beneficiaries?
  2. What percentage of people use their stove for all cooking and heating functions?
  3. Why do people not use the stove for all cooking functions?
  4. Has the production and distribution programme reached its targets? Reasons?
  5. Has the extension and/or marketing programme reached its targets? Reasons?
  6. What is the impact of the stove's introduction. Are the objectives of the programme being met?
  7. Have there been some unforeseen impacts of the programme? What are they?

Alternative methods of data analysis

Participatory methods

Role playing

One method that has been successfully used in Africa to interpret the information collected by members of the stove group is role playing. There are a number of different techniques that have been developed. The following is an example of how a role playing session could be run.

The same role play can be refined and presented as a play to villagers to help enlist support for the stove group's work.

Verbal interaction method

Members of the stove group are provided with a cassette player (or use their own) and a chairperson is chosen to help direct the discussion. All participants start their recorders and the chairperson asks individuals to report their findings and suggest methods of improving the stove and/or the extension programme. No conclusions are drawn during the first meeting. The group chooses to meet within a week and agrees that each individual will listen to the tape a number of times before the next meeting. At the next meeting each member is requested to give their interpretation of the data that has been presented by the others. The chairperson then tries to draw conclusions from the ensuing conversation and the group produces some type of graphic output.

Conventional approaches to data analysis and reporting

The results of monitoring can also be analysed numerically and graphically. In the case of stove testing it is essential that the average of the tests and the standard deviation be calculated for both the new and the old stoves. A special test, known as the student t-test can then be applied. This test will help determine if the difference in performance between the 2 stoves is statistically significant. Details of this test can be obtained from a statistics textbook. The student t-test can be done on a calculator containing an appropriate programme or on a computer. If calculator processing is not used, information from the sample surveys must first be tabulated, and the frequency of a particular observation calculated by hand. For example, a breakdown of the types of repairs that have been done on new stoves could be presented as it is in Table 3.1.

Type of repair # of stoves Percent (%)
Repair not required 53 29
Completely repaired 34 18
Partially repaired 15 8
Not repaired 86 45

3.1 Number of stoves repaired

More detailed tables can also be constructed. For example, in the Nepal Programme the managers wanted to know, if during the initial field test phase, there was a relationship between acceptance of the stove (measured by how often the stove was used) and the village where the stove had been introduced. Table 3.2 on the next page was prepared and a special type of statistical analysis, a Chi Square analysis, was performed (the interested reader is referred to a standard text on statistics for further information).

Stove use
Village Exclusive All meals Most Meals Infrequent Not Used Total
Chapagon 19.7% 38.0% 25.4% 15.5% 1.4% 37.2%
14 27 18 11 1 71
Sitapaila 18.7% 4.0% 49.3% 0.0% 28.0% 39.3%
14 3 37 0 21 75
Lamatar 20.0% 28.9% 6.7% 4.4% 40.0% 23.6%
9 13 3 2 18 45
Total 19.4% 22.5% 30.4% 6.8% 20.9% 100%
37 43 38 13 40 191
For each row, percents are on top and counts are below.
DF = 8, Chi Square X2= 73.52

3.2 Level of acceptance for three villages

The Chi Square analysis indicates that the stove had much greater acceptance in Chapagaon. Previous monitoring had revealed that the main difference between this village and the other two was that in this village fuelwood was scarce. The programme thus concluded that there was a much greater chance of stoves being accepted in villages where fuel was scarce.

The clearest way to present data is graphic. The histogram on the next page (Figure 3.3) clearly shows the number of stoves being used (as compared with the number introduced) in 3 different socioeconomic groups. It can be seen that the middle class people have higher rates for both stove adoption and use than the lower and upper class people.


3.3 Stove usage versus socioeconomic group

3.4 is more complex. It summarizes the major complaints that different groups of people have about the stove. The lower status groups, where time is very scarce, find that the stove's major fault is that it cooks too slowly. Alternatively, high status group are more worried about the stove's appearance.


3.4 Graph of user complaints for different socioeconomic groups

Figure 3.5 indicates that the programme is not meeting its production targets. This graph can be easily understood by both community members and programme staff and can be prominently displayed.


3.5 A plot comparing monthly stove construction targets with actual rates of construction

Figure 3.6 is a Pie chart that illustrates the charcoal consumption patterns of people who have and who have not purchased the new stove. The data indicates that most adopters are people who use charcoal as their main source of fuel. Non-adopters use primarily other sources of fuel such as gas, kerosene and wood.


3.6 Charcoal consumption by new stove adopters and non-adopters


BACK TOP TABLE OF CONTENTS BOTTOM NEXT

4: Collecting and Storing Data

Four main methods of data collection have been used by local people, programme personnel involved in monitoring, researchers and external evaluators:

  1. direct measurement (s)
  2. participant and non-participant observation
  3. case studies
  4. surveys

These methods are complementary and can be used to answer the questions that were posed in previous chapters. As has been demonstrated in the previous sections different combinations of techniques are used during different phases of programme implementation. The type of information that is required will affect the choice of method. Each requires pretesting by experienced people before the technique is used on a large scale. Pretesting involves testing of the measurement techniques, creation of observation sheets, check lists, interview lists or questionnaires to determine if:

  1. there are ambiguities in the questions,
  2. the observations or questionnaires can be administered by field workers,
  3. the data are easy to process,
  4. the measurements are replicable when carried out by different people.

In this section the techniques and requirements for each method are outlined, giving specific examples. The techniques that are most common and/or most appropriate for use in each programme phase are also identified. The difficulties associated with the implementation of each approach are highlighted.

Methods of Data Collection

Direct Measurements

Local participants, designers and field workers must have accurate information on fuel consumption, quality of production, installation, stove deterioration over time, amount of ash build up on the stove, levels of pollution, and innovations that users have made. Measurements can be taken during the survey or as a separate exercise, but it needs to be remembered that direct measurements are intrusive and in some cases can be either offensive or induce fear, especially if people are occasionally harassed by other groups or government officials. It is important that people receive detailed explanations of the study's purpose and the method of the observation. Taking direct measurements is time consuming and enumerators require considerable training.

Care must be taken when measuring fuel consumption in households, if data are to be both representative and accurate. The sample should consist of both stove users and non-users. Recent experiences in Kenya and Sri Lanka indicate that the following procedure can be used to get reasonably accurate results (for more extensive discussion on woodfuel measurements see Woodfuel Surveys, FAO 1983):

  1. Place measured wood stocks and residues in houses of users and non-users.
  2. Ask households not to lend or borrow any fuel during the test.
  3. Measure fuel either every day or, if resources are limited, at the end of each week.
  4. Take measurements before introducing a new stove to the user group and then three, six and twelve months after the introduction.

There are two requirements when undertaking measurements. The instruments should be accurate and reliable, and the sheets to record the values must be clearly designed and contain sufficient information to assess variations in the results. Careful instructions must accompany the sheets and a second person should check the measurements in the field. Standard sheets and procedures have already been devised for measuring fuel consumption (Appendix 9). When measuring levels of pollution the procedure developed in Nepal and that of the East West Center (1983) can be used. When measuring the stoves, create a sheet that has both a dimensioned sketch of the stove and spaces to provide the desired dimensional information. The picture can be used to illustrate the exact position of cracks, the position and quantity of soot, and the places where the dimensions or installation are faulty.

When assessing quality of production, it may be necessary to take a random sample of stoves and measure their dimensions. These can then be compared to the project standard. Deviations from correct measurements can then be calculated. Producers should be visited regularly and changes in quality can be assessed over a period of time. Measure changes in the dimensions of the stove in a representative sample at regular intervals for a period of at least one year. During these visits observe changes in kitchen layout, technical changes made to stove design, recent maintenance or repair. Field trip notes should always be filed and continually processed.

It is important to develop special books or registers that allow easy data entry and retrieval. The number of stoves produced, distributed and installed, the number of training sessions or community meetings, the number of new producers and distributors and project expenditure should all be recorded in registers. This is the operational data that are vital for determining if targets are being met. These books should be collected at regular intervals so that the data can be collated in a central registry.

Participant and non-participant observation

Participant observation is a technique used by researchers. A researcher lives with either one or a small number of communities. Over a period of weeks or months the researchers try to integrate into the community so that they can get unbiased detailed information through observation, structured, and unstructured interviews. They usually collect and cut wood and cook, gaining a greater understanding of the issues that face the stove user. The information they obtain can be on local values, politics, social structure, factors affecting decision making, attitudes to development issues, status of women, time allocation, cooking and fuel gathering practices. At the beginning of research, develop a checklist of the data that are needed. This information is collected over a period of time that the programme managers must specify. The longer the period, the more in-depth and wide-ranging the collected information can be.

The researcher sets up a filing system for recording information from key informants, and details of other events. Key informants are people who have a particular knowledge about a subject or who are opinion leaders in their community. The researcher, serving as facilitator, may also get groups together to discuss certain issues related to use and adoption of the stove. The research may be more or less time-consuming, but it can provide essential background and back-up information. It allows in-depth analysis of a particular event or result, but does not allow generalisation. It may help managers determine why a particular district has a low or high rate of adoption, why the community is or is not actively involved in the programme, or why fuel savings is lower than expected. It can also provide detailed information on the programme's impact. The sex or age of the researcher can bias the data (e.g. a young man may be considered too young, a woman may not be allowed to attend council meetings. Men may not be able to talk with women or to actually use a stove).

Non-participant observation is a relatively non-intrusive method of data collection that can be done without the active participation of the people being observed. Examples of non-participatory observation include:

  1. recording the number of people examining or purchasing the stove and the method of selling the stove;
  2. walking through the artisans' working area or through villages and observing the number of people producing the stove and the methods of production;
  3. observing the pattern of interaction of the community and field staff at meetings.

Observation can yield an instant picture of what is actually happening. Additionally, the chance that the subjects' are changing their behavior patterns due to the presence of the researcher diminishes. The information is however, susceptible to observer bias and the amount of data that is collected can be small compared with the resources required. Non-participant observation is an extremely good method to use to validate and complement information gathered from surveys.

Case studies

Case studies use participant observation to do an in-depth study of a few households. Local adults or school children and/or researchers live with a few families over an extended period of time and obtain detailed information on patterns of fuel consumption, changes in time allocation and patterns of economic activity. Local or external researchers can gain access to a number of households that are representative either of different socioeconomic groups, or of different cooking patterns. They work with the women over a period of at least one month, helping carry wood or cook food. During this time they measure consumption, time allocation, observing cooking patterns and decision making within the household. Additionally, they have detailed conversations with the women of the households. From these case studies a preliminary hypothesis is developed that can be tested in the larger surveys. The tabulated results from the case study can help determine if there is the possibility of major error in the survey.

This method suffers the same drawbacks as participant observation. The sample may not be representative and a considerable time period is needed to collect and analyse the information. However, the researcher will be able to provide the programme manager with real insights into why certain patterns of usage and adoption are being observed in a particular area.

Sample surveys

These are the most popular means of obtaining information in stove programmes. Surveys are used to collect information on rates of adoption, usage and impact. A survey involves interviewers asking representative individuals and groups of people (a sample) a series of questions in a particular area.

A sample is a representative subset of a group of things, people or districts known as the population. A population can be divided into units, either people, households or villages. The list of sampling units is called the sampling frame. In the early stages of a stove programme all the households can be monitored. As the programme introduces more than 200 stoves and expands into new areas it is not possible to do detailed surveys of every households. That is when sampling becomes necessary.

If you have not received training in sampling techniques seek help.

Choosing a sample is not an easy task. When cooking and fuel use patterns, income levels, religious affiliation etc. are very diverse, a careful selection process must be devised. In this case the input of a trained social scientist or a statistician is required. However, for those who have already received training in sampling, a number of practical examples follow. For this method to work, a distribution register must be maintained.

EXAMPLE 1 The sampling method used by the Department of Community Forestry in Nepal.
Step 1: Construct a complete list of all households who have received improved stoves, making separate lists for each year of stove distribution. Number sequentially each household on the list beginning with 1.
Step 2: For each year, divide the total number of households by the total number to be surveyed, to obtain the Sampling Interval (SI), e.g. 693 households divided by 150 sample size equals 4.62 SI. The decimal points should be retained to two or three places.
Step 3: Select a number (R) between 1 and the sampling interval (SI) from a random number table. The household whose serial number corresponds to this random number is the first household in the sample, e.g. R = 2.
Step 4: Successively add the sampling interval to the first random number and round off the results until the end of the list is reached. Each of these rounded off numbers corresponds to the serial number of the household selected, e.g. R = 2, SI = 4.62.

Serial Number
First household selected 2 (= R)

7 (2 + 1 x 4.62)
11 (2 + 2 x 4.62)
16 (2 + 3 x 4.62) etc.

EXAMPLE 2 Sampling in an Urban Environment


Grid System

  1. Initially you may have to create a map of the area to be surveyed.
  2. If there are regular blocks of houses, assign a number to each block. If there are not, draw a grid over the entire area. The size of the grid will depend on housing density and the total area to be sampled. In a high density area each grid should have at least fifty houses. Number each grid.
  3. Determine the total number of grid areas/blocks to be surveyed and the number of houses in each area.
  4. Use a random number table to choose the areas to be surveyed.
  5. Number the entry points into an area and flip a coin to determine which entry point to start from.
  6. Use a coin to decide whether to survey a house (e.g. you decide heads to interview. At each house you flip the coin and if it is heads you interview). A dice can be used to limit the sample in urban areas. A house is entered if the dice lands with 1 upwards.
  7. If the person does not want to be interviewed or is away go on to the next house and flip the coin.

Developing questionnaires

There are two types of questions that can be asked:

open ended e.g. Does the stove do its job or not ?
and closed e.g. Do you use your stove for cooking:

  1. all of your meals;
  2. some of your meals; or
  3. none of your meals?

In the latter case people are asked to choose from a small number of alternatives, thus providing fairly precise information that is easy to analyse. In the former, people are asked for an opinion and the possible answers are very diverse. The answers from open ended questions can be used to develop closed ended questions. They also provide much more insight into the differences in opinion between households. Usually both types of questions are included in a survey since they are complementary. Closed questions are easy to ask and process, but they limit the type of information that is obtained. Open questions give useful information, but in a form that may not be easy to process. Questions should follow in a logical order to maximize the amount of information that is elicited. They should fit the local perceptions and use the people's understanding of the way things function.

It is important not to ask direct questions. Ask indirect questions that will elicit the desired information. For example, instead of asking 'does your stove save fuel?' ask

  1. 'When did you last use your stove?' (check the answer against the stove temperature; a hot stove means it has been used). If the answer is one or two days ago, then ask:
  2. 'How does your stove work? What kinds of problems have you had?' If the answer is none, then you may ask 'What changes has the stove made to your kitchen?' or 'What do you like/ dislike about the stove?'

The quantity and quality of the data will depend on the effectiveness of enumerator training, motivation, and the rapport that enumerators can establish with the people being interviewed.

There is a tendency to ask too many questions. Much of the data is never analysed. Experience suggests the following procedure:

  1. Determine the precise objectives of the survey (e.g. to determine why stoves are not being used in a large percentage of households).
  2. Talk with field workers and the local stove monitoring group to determine what they think are the main problems in the area.
  3. Develop hypotheses based on these discussions, e.g. the main reason for low stove usage is breakage.
  4. Base the questionnaire on these hypotheses and test it in the project area using approximately 20 randomly selected households.
  5. Determine which questions are unsuitable because of ambiguity, offensiveness or any other reason.
  6. Make alterations and design the survey so that it is easy to process.
  7. Make most questions closed ended but leave space at the bottom for users' /producers'/enumerators' comments and observations.

Data collection in different programme phases

Each method that has been mentioned gathers specific types of data. However, the most appropriate data collection techniques can vary based on the stage of the programme and the types of information that are needed.

Collecting baseline data

In Sri Lanka, field extension workers have generated baseline data by recording users' observations, discussions with key informants and group interviews. They have also examined locally available documents using the following procedure: Before visiting a district :

  1. Obtain aerial photographs of the proposed field area if possible.
  2. Identify areas that are ecologically different and, if possible, different patterns of housing within those areas. Visit houses in each different field area.
  3. Obtain available literature, census and agricultural data on the field area.
  4. Identify regions within the field area where fuel use patterns, cooking practices or socioeconomic status are different.

    After arriving in the district :

  5. Hold a meeting with local organisations to learn how they perceive local differences. Select specific areas that represent the different local types to visit.
  6. Draw up a list of the information that is required (see Appendix 3).
  7. Spend one or two days in each area. Visit houses at random and record detailed observations. Ask a few women and men if they would agree to be interviewed. If possible, interview people with specific relevant knowledge (stove builders, fuel sellers for example). You can interview people from particular areas in groups during a night-time meeting. The types of houses, size of agricultural holdings and number of people earning a cash income can all be used as key indicators of socioeconomic status.

Initial field testing phase

Four main data collection techniques can be used by community groups, extension workers and designers.

Log book system:

At the beginning of the programme, local organisations can establish a very simple log book system. This records who has received the stove and when, how many people participated in its construction or installation, how much was paid by the recipient (or food provided to workers), how much time was used for construction. Appendix 5 contains one example of a log book design.

In some programmes a volunteer working with the local stove committee or a full-time stove worker, makes regular visits to the households. During these visits the worker conducts an informal interview and observes the household. These can be recorded in the log book. Depending on the skill of the interviewer, considerable information can be obtained by talking generally about needs. This can be recorded at the bottom of the log book page.

Minute record system:

A local organisation can also establish a simple system to record stove related activities. This can consist of records of minutes of meetings. An example of a simple minutes book is given in Appendix 5.

Diary system:

A diary of local events can be kept. For the two villages in a project funded by the Ford Foundation in North India, the project leader kept a detailed diary of people's sense of changes in fuel consumption, cooking time and kitchen appearance in the two villages where the field programme was initiated. She initially conducted boiling water tests by recording time water took to boil and the amount of water boiled by identical quantities of wood in traditional and new chulas. In 14 households (seven with old and seven with new stoves) the weight of wood used over a six-week period was recorded by literate children under the project leader's close supervision. Tests were assessed to determine if the new stoves were superior to the existing ones.

In Nepal, RECAST informally monitored stove acceptance in one village by frequently visiting households to discuss any problems. The stove programme manager kept a diary of events. Approximately 15 households completed water boiling tests. At the same time, baseline data was collected by questionnaire from households both with and without the new stove. Follow-up water boiling tests were done after stoves had been in use for six months.

Controlled testing method:

In programmes with considerable resources the following data collection method can be used.

Approximately 50 households are chosen at random to participate in the initial field test phase. A known weight of fuel is given to each household. Consumption of fuel is measured one to two weeks before introduction of the stove and one to two weeks after. Socioeconomic data on each family is collected during this testing. Details of the recommended procedure are provided in Appendix 8. In some cases it has been difficult to follow this procedure due to cultural values. Instead of measuring the fuel, it is possible to question the amount of money that is spent on fuel, frequent cooking tests can also be done. Again, tests are carried out six months after the stoves are installed.

As well as testing, the stove team can also monitor response to the stove. Either a log book or a diary can be used.

Expanded field testing phase

Surveys are commonly used for data collection in the expanded field test phase of field programmes.

In Sri Lanka, a questionnaire was administered to 200 of the 500 households that had a new stove. Members of the stove unit, who were chosen from surrounding villages (not trained enumerators) administered the survey. Initially, the questionnaire contained 30 separate observations /questions. Careful pretesting indicated that only 13 observations /questions were needed to obtain reliable information on patterns of usage. The questionnaire was very simple, yet collected vital information. Much of the information was obtained from observation and most of the questions were closed ended. This questionnaire is reproduced in Appendix 6.

Checks were built into the questionnaire. For example, if the interview was carried out after breakfast, the new stove was cold, the traditional stove had not been removed, and the user answered that the new stove was used for all meals, the enumerator tended to discount the reply. Data on the quality of stove construction was collected. This provided a check on users' perceptions of savings since a poorly built new stove was often less fuel efficient than the pre-existing stove. In some households, people would indicate reasons for satisfaction or dissatisfaction with the stove. This was noted at the end of the questionnaire under the 'comments' heading.

Data storage and analysis of information from this short questionnaire was easy. However, while it did provide an accurate picture of design problems and rates of usage, it did not provide in-depth information on why certain patterns of usage were being observed in certain areas.

If possible, this type of survey can be followed by in-depth interviews. This involves choosing four to eight households that represent both a range of usage patterns and socioeconomic groups. A trained social scientist can spend a few days with each household building up a detailed picture of cooking practices, time allocation, decision making practices, patterns of fuel use, etc.

Some programmes, e.g. Nepal Community Forestry Development Programme, have more resources available and thus the surveys are much more complex than those carried out by the smaller programmes. An example of a detailed survey is presented in Appendix 7. Questionnaires are developed either by the Monitoring and Evaluation Unit or, in some programmes, by the manager. They are first pretested in 20 - 90 houses to determine which questions are redundant, ambiguous or poorly worded.

The Nepal questionnaire tried to determine the amount of time a stove was being used, how much fuel it consumed, and why a stove might not be used. Since detailed baseline data on particular households were collected at the beginning of the field programme, usage patterns could be related to such variables as household size, stove type, ethnic group, traditional stove type, stove condition, frequency of maintenance, accuracy of installation, pot size fit, perceived fuel savings, cooking time, convenience, promotion and extension programme effectiveness and socioeconomic status.

Checks similar to those in the Sri Lanka questionnaire were built in. In addition, a question on 'willingness to purchase a replacement' was included to help determine the acceptability of the stove. The questionnaire was designed so that the data could be entered into the project's microcomputer quickly. It was administered by trained and experienced enumerators.

Experiences in Kenya and Indonesia have shown that it is not very cost-effective to try to use one questionnaire to gather both baseline information and information on patterns of usage/ acceptability. Such questionnaires are long, and can be difficult to interpret. They may be poorly administered and the answers may be suspect. Because the questionnaires are long, few are administered if funds and time are limited; hence, conclusions may be unreliable. In most cases, the time taken to analyse the data has been so long that it has been of little use to project management.

Expansion phase

In the expansion phase, four programme elements can be monitored: the quality of extension in programmes that introduce stoves that must be installed by stove builders; stove production; stove sales in the market place (such stoves can be portable or may need to be installed by the household. The fourth exercise determines why people are or are not adopting a stove and the impact of the programme.

Installation form

Appendix 5 a copy of an Improved Stove Monitoring Form that was developed for the Division of Community Forestry Programme in Nepal. It was used to monitor the quality of stove installations. The stove consists of three ceramic parts held together with mud. The stove is placed by a local person and the district promoter (stoves extension officer) comes to visit the house after it has been installed. The data that are collected provide information on the effectiveness of the extension programme (e.g. has the extension officer visited the house on time and do people know how to clean the stove). A check list can help the extension worker. During the first visit the promoter shows how to use and maintain the stove. The promoter then returns three months later to determine if there are any problems, either with the stove itself or with the users' methods of operation. These sheets are regularly reviewed by the project engineer, manager and/or monitoring and evaluation units of the programme.

Information from this monitroing form can help determine the effectiveness of the extension effort. Programme staff should sometimes review the information that is collected with the extension workers, both to check the information and to increase extension workers' incentive to take time and properly fill in the forms.

Production unit monitoring

Various methods can be used to monitor the output from the production unit. In Nepal, a detailed time and motion study of the production process was carried out for two production units. The study was done for all units at the beginning of production and three and six months later. A simple book was created to record the following:

Date # made # stove workers Kgs clay # fired # rejects # sold





















Stoves sales monitoring

A wall chart can be used to record stove sales. Information can be collected in a book and immediately transferred to the chart which has axes plotting the number of stoves sold at different locations each month, or the price of stoves quoted by different vendors each month. Experience has shown that prices fluctuate with supply and demand as well as location. Alternatively, cards can be kept on each sale. Information on sales, prices, and reactions to stoves from different consumers can be recorded on these cards.

Patterns of adoption and impact

To monitor patterns of adoption and impact on households, collect data over at least two years. Select at least two representative villages (or clusters in an urban setting) in each area where the stove programme is to be implemented. Undertake a detailed socioeconomic survey that examines health, fuel consumption and cooking patterns. Stoves can be introduced in one village and not in the other so that detailed measurements, observations and interviews can be carried-out over the two year period to determine the changes resulting from adoption of the new stove.

In particular the programme can try to determine:

  1. whether the stoves have spread from one village (cluster) to the next and why/why not?
  2. what aggregate changes in fuel consumption, the type of fuel being used, cooking time, smoke and hygiene related diseases, the kitchen, patterns of labour allocation, levels of participation in existing development programmes, or the amount of new programmes can be attributed to the introduction of the stove?

It is very difficult to specify a type of questionnaire that can be used, large scale surveys are not often done. A monitoring exercise that one programme used illustrates how a questionnaire could be structured and designed. This is presented in Appendix 8.

Data storage

A great deal of data will be produced through a comprehensive monitoring programme. Some of this data can be used immediately, other sections can be stored for use at a later date. Often, data processing and storage is rather haphazard. This results in both loss of data and difficulties in analysis. A range of manual and microcomputer storage techniques are available to programmes at relatively low cost.

Card system

This is a technique popular with anthropologists. Data is normally segmented, coded and stored in a master file. The master file can be divided into villages and subdivided into households. Key information can be placed on the card:

Name: Type of stove:
# of stoves: Date installed:
# of children: Dates of extension visits:
Ethnic group: Workers who visited:
Indicators of socio-economic status: Date when stove stopped being used:
Date repaired:

On the back of the card a series of reference numbers that refer to other information about this family can be listed.

Information Card #:
Perceived costs and benefits (11):
Fuel use data(15):
Patterns of usage(14):

Data on these subfiles will often be coded to facilitate analysis. For example, consider the card 14 below:

14. Usage patterns Village #:
House #: Usage pattern:
1. c
2. e
3. a
4. d

a=all of the time....e= not at all

Some data is aggregated before being coded. For example, socioeconomic status may consist of a combination of indicators such as housing, land holdings, occupation. A family might have a high status if it had more than five acres of land or a person working in the government. Medium status might include a family who owned a house with three rooms and land holdings of three to five acres.

Research book

Some programmes have used a large ledger. All of the data on each family can be coded and then placed in a ledger, as in the example below:

Date Name House # # People Usage rate Fuel savings Smoke levels
1/10/83 Min 8 6 C E A
3/02/84 San 6 5 D A B






etc.

Computer storage

If the resources are available, data can be stored in relatively inexpensive microcomputers. A special statistical or data management programme can be purchased. This programme allows files to be established for households in each village or district. The data (or field) for each household are held in its record. The programme allows you to collect data on a specific topic (e.g. acceptance) from each record. It also permits searching for a particular record(s) that contains needed data.

Two types of microcomputers can be used. One uses MS-DOS as the operating system (e.g. the IBM PC), the other an Apple Macintosh. The Macintosh is a more expensive computer but is easier to use and can produce much higher quality reports than the MS-DOS machine. If the programme personnel have not used Microcomputers for statistical analysis it is strongly recommended that they either pay for a professional to set up the system and train the staff, or hire an experienced programmer.


BACK TOP TABLE OF CONTENTS BOTTOM NEXT

5: Training for monitoring

Who is trained and what subject matter should be included?

Managers and designers

If there is to be a general acceptance and knowledge of the monitoring, senior staff working in the M & E unit, or designers and programme managers, should receive training.

The training course can examine:

  1. the importance of a participatory approach in a stove programme,
  2. planning and management of monitoring programmes,
  3. integrating monitoring into programme management,
  4. methods of data collection, processing and analysis,
  5. training methods for field staff and enumerators,
  6. communicating results.

Users and producers

User and producer training is essential if monitoring is to be effective. Training materials could discuss:

  1. the reasons for introducing and monitoring of new stoves,
  2. the good and bad features of stoves currently in use,
  3. basic stove design and testing principles,
  4. what makes a stove work properly,
  5. how monitoring is carried out,
  6. effective communication with programme staff and users.

Enumerators and testers

Enumerators can be involved in the initial survey phase and monitoring in the expansion phase. Their course can examine:

  1. the design and use of a new stove,
  2. why a stove does not work properly,
  3. how to test stoves,
  4. interviewing and observation methods,
  5. use of survey questionnaire.

Field extension officers

Field extension officers would probably be responsible for monitoring:

  1. production of stoves by artisans and other field personnel,
  2. installation of stoves,
  3. implementation of training or promotion campaigns,
  4. implementation of community development programmes.

The type of training course that is developed will depend on the responsibility of the officers. For officers who are involved in small programmes training can teach:

  1. how to design stoves and establish indicators for monitoring programmes,
  2. how to implement monitoring programmes with community groups,
  3. simple methods of data collection, storage and analysis,
  4. methods for communicating results to all sections of the community,
  5. methods for training community members in monitoring.

The Division of Community Forestry Programme in Nepal has developed a particularily effective training course for field workers that includes a monitoring component (See Appendix 10). In this programme, district stove officers are responsible for production, distribution, and installation of the stoves, and for liaising with the community leaders and other organisations. They are also responsible for monitoring the performance of the potters and installers, and for collecting data for the monitoring and evaluation unit, which is situated in the head office. Before they commence the job they receive training in stove design, testing, production and installation, and instruction on methods of data collection and analysis. The head of the M & E unit and the consultant attached to the unit provide the specific inputs on collection of baseline data and monitoring. The topics include:

  1. a review of kitchen and stove types, fuel use and cooking practices in Nepal
  2. stove choice and the way it relates to cooking practices and kitchen and fuel types in Nepal
  3. the rationale for collecting base-line data. How to collect data using the 'traditional stove use survey sheets'
  4. the art and science of surveying
  5. how to use the 'improved stove use survey form'
  6. the use of sampling and analysis of completed surveys
  7. how to complete a stove distribution register and follow-up register

When is training done and who teaches the course?

Enumerators and designers should receive training at the beginning of the survey phase. The course can focus on how to collect baseline data and choose sites for initial trials. It can be conducted by consultants from national research groups or, in the case of national programmes, by experts in stove programme assessment. Before the initial field trials programme managers may train community groups or individuals to observe and keep simple records of user reactions to stoves.

Many programmes have trained extension officers to collect more detailed information as the field trials proceed. The data record users' reactions, stove performance, production quality and quantity, and installation. Some programmes have also provided further training for senior programme staff and members of the M & E unit, on survey techniques, participant observation and measurement techniques for the expansion phase. During the expansion phase extension and community development workers have received follow-up training, either each year, or before a major data collection exercise has been implemented.

How is training done?

Training is carried out both "on the job" informally and in classes or formal training sessions.

Formal training is preferred when more than 2 or 3 people require instruction. Whether the training is formal or informal, the instructor should prepare for the sessions.

Training involves three processes: preparation, instruction and checking. Preparation involves focussing on desired results, planning logical learning stages and deciding the key concepts that need to be conveyed. In introducing each stage the instructor outlines the objectives and the key points. Each key point is accompanied by a practical demonstration or a question. Trainees complete each practical exercise and the instructor corrects errors. Trainees are encouraged to help each other and role play is used to highlight the most important aspects of different monitoring procedures.

Field work is an essential part of the training programme. In the initial survey phase, and during senior staff training, it is important that participants design a questionnaire, pretest, and evaluate it under the supervision of a trained social scientist.

Summary

For monitoring to be effectively done by all members of the programme, participants need to receive basic instruction in:

  1. the role of monitoring in programme management,
  2. the methods to be used in the programme,
  3. the role that they will play in the monitoring programme.

During implementation of the programme, various groups of people could receive specific training related to individual monitoring activities. If resources are available, all staff should attend refresher courses.

Effective training emphasizes "learning by doing". Field exercises are an essential component of the training.


BACK TOP TABLE OF CONTENTS BOTTOM NEXT