Completion evaluation

The IFAD Office of Evaluations and Studies carried out a Completion Evaluation (CE) of the IFAD-financed North West Agricultural Services Project (NWASP) in Armenia from February to June 2001. The IFAD loan to finance the Project was identified in 1994, approved in December 1997. The loan became effective in April 1998 and was scheduled to close in December 2002. However, due to rapid implementation, Project activities and disbursements were more than 90% complete at the time of the evaluation.

The Project was subject to a Completion Evaluation because it is part of an IFAD pilot programme for direct supervision. The Evaluation gave IFAD the first opportunity to analyse the experience of direct supervision. It also provided an opportunity to draw lessons in both water management and rural finance for reference in the design and implementation of other IFAD projects in Armenia and elsewhere in the region.

The evaluation

The IFAD approach to evaluation included the creation of a Core Learning Partnership (CLP) of key project stakeholders that helped to set the objectives and key questions for the Evaluation. The CLP reviewed the draft Evaluation Report and participated in a Workshop held in Tsakhkadzor in August 2001 where it concluded an Agreement at Completion Point that detailed those recommendations and follow-up for which there was a consensus.

The objectives set for the Completion Evaluation were to: (i) learn how, and to what the extent, the project achieved its overall objective to improve the living conditions of the target population; (ii) determine the impact that direct supervision of NWASP by IFAD has had on NWASP and on IFAD; and, (iii) provide feedback on NWASP project outcomes that can help partners associated with the future Agricultural Services Project successfully achieve their objectives. The key questions posed by the CLP were:

• How did the Project change the living conditions of the target population in terms of income, production and food security?
• To what extent were the designs of the various components relevant to the needs of the target group?
• How effective were implementing agencies in reaching their physical objectives?
• How efficient were the implementing agencies in achieving the maximum level of benefits with the resources allocated to them?
• To what extent can the benefits of the various components be sustained after project completion?
• What were the costs and benefits of Direct Supervision of this project by IFAD?

With respect to the dissemination of Evaluation findings and recommendations, the CLP placed a high priority on using the Evaluation to create a greater awareness of the project outcomes and lessons within higher levels of Government.

A participatory approach was used wherein stakeholders, including several project beneficiaries and implementing agency staff, served as the Evaluation Team working in the field under the guidance of a Team Leader. In addition, the evaluation included a Socio-Economic Survey that repeated interviews with the same 699 households interviewed for the Socio-Economic Baseline Survey in August 1998 and again in late 1999.

Altogether, the sources of information for the evaluation included: (i) more than 300 beneficiaries, leaders of communities, and grassroots organisations in 48 communities interviewed during evaluation fieldwork; (ii) the 699 households interviewed for the formal socio-economic survey; (iii) key informants from implementing agencies interviewed by the Team Leader; and, (iv) IFAD staff and consultants responsible for project supervision who responded to a questionnaire or were interviewed.

At the end of the fieldwork, participants in the evaluation and agency staff rated the effectiveness of the evaluation. In each marz work concluded with a briefing for officials and community and implementing agency leaders presented by beneficiary members of the Evaluation Team. A debriefing / wrap-up seminar was held in Yerevan attended by high-level Ministry officials on 6 April 2001. At the seminar reports were presented on each project component by beneficiaries, and on overall preliminary conclusions by the Team Leader. Finally, the evaluation processes concluded with a Workshop held in Tsakhkadzor in August 2001 with the drafting of an Agreement at Completion Point that includes those recommendations and follow-up upon which there is a consensus.

Back to Top

---

The project design

The goal of NWASP was ‘to improve the living conditions of the target population in terms of higher levels of production, income and food security'. It operates in three Marzer: Aragatsotn, Lori and Shirak in the north-west. The Ministry of Food and Agriculture (MOFA) is the project executing agency. It established a Project Co-ordination Unit (PCU) based in Yerevan that implemented the project through six agencies: Water Management Services (WMS) section of the Irrigation Rehabilitation Programme (IRP), the Agricultural Cooperative Bank of Armenia (ACBA), an international NGO called the Armenian Technology Group (ATG), and an Armenian NGO, SHEN.

The estimated total project cost was USD 13. 4 million of which USD 12. 9 million was funded by IFAD and USD 0.5 million by the GOA. As at 1 March 2001, approximately USD 12 million had been disbursed. Project implementation activities were scheduled to end in June 2001, approximately 18 months ahead of schedule.

The target population numbered about 335 000. It was composed largely of small farm households with an average farm size of 1.3 ha. Many of these had been part of the commune/state farm system before the privatisation of farming land. Some were headed by individuals allocated plots of land after being dislocated from industrial enterprises that closed down when Armenia became a separate republic. Benefits were expected to reach some 9 500 households about 13% of the total. Activities in the community development component were to be directed to the poorer villages. In the Credit component the aim was to assist to women take up to 30% of loans.

The project had four components:

Irrigation rehabilitation: to rehabilitate irrigation water supply and control structures; and help improve management, operations and maintenance (O&M) of the irrigation systems through water user associations

Rural credit: to support ACBA in increasing its lending services to farmers in the three marz, including the provision of capital and on-lending resources, building and equipping branch offices and development of village associations (VAs) as shareholders in ACBA to organize borrowers at a village level

Agricultural and livestock development: to produce good quality seed and provide veterinary services for livestock production

Community development: to assist the poorest villages to identify, plan and implement their own priority activities using participatory techniques.

Back to Top

---

The key questions

How did the Project change the living conditions of the target population in terms of income, production and food security?

Findings with respect to income and living standards from the survey were difficult to interpret. The Survey found little difference between beneficiaries and control group households in the increases in the value of their average total income (cash and own consumption) between August 1998 and March 2001. At the time of the survey, farmers were experiencing drought conditions and approximately 80% of beneficiary households reported incomes below the national poverty line. However, they fared somewhat better than control group farmers, 90% of whom reported to be below the poverty line. Both groups have experienced a sharp decline in living standards from the time of the repeater survey in October 1999 when only 6% of beneficiaries and 18% of control group members were below the poverty line.

The survey found significant changes in the agricultural production levels of households that participated in NWASP compared to those from the control group who did not participate in the project. For example, for a typical beneficiary household output in cereals and potatoes increased by more than 66%. Whereas, output for control group households stayed the same or fell with respect to those two crops. Outputs of vegetables, fruits and fodder were also higher for beneficiaries than for control households. Overall, the total value of their output was 30% higher than the value of agricultural output of the control group.

With respect to food security, the percentage of households with a three-month supply of food in storage fell for both the beneficiary and control groups. This was largely due to on-going drought conditions and seasonal difference in the period during which data was collected. However, the drop in beneficiary households with such stocks was only 3% while it was 20% for the control group. Food intake has also become more regular among beneficiaries. At the time of the baseline survey 19% of households reported skipping daily meals. That figure has now dropped to 10% among project beneficiaries, and it is 17% for the control group. The survey showed that 66% of women and children consumed three meals per day in the four months preceding the survey, while only 50% did so at the time of the baseline survey in August 1998.

To what extent were the designs of the various components relevant to the needs of the target group?

The Evaluation found that beneficiaries were largely satisfied with the designs of each component, subject to constraints imposed by funding. Communities indicated that inadequate irrigation resources were the major constraint to improved living conditions. Moreover, irrigation improvement had widespread relevance, providing benefits in a relatively equitable fashion to a cross-section of the community. Because design and implementation was guided by community groups it provided a strong link to community needs.

The same held true for the community development component. The processes designed were relevant. Moreover these processes allowed communities to demand and effect a shift in the types of activities selected the project would support. Yet, the shift created by the new activities chosen by community Civil Action Groups was not entirely positive. It re-allocated project funding from public to private goods. The evaluation team observed that in these cases most of the direct benefits from the project went to a very small number of beneficiaries who owned agro-enterprises that received grants. The responsible NGO argued that the indirect benefits to the community from the new enterprise in terms of economic activity and employment offset this seeming imbalance.

The credit component was relevant to the needs of many members of the target group. However, it was implemented in such a way as to be accessible to very few. Limited funds were effectively rationed by relatively stringent collateral requirements. In this way the project helped build a sound financial institution that, in future, is likely to reach further down to smaller, higher risk farmers. However, during the project implementation period this group did not benefit from project-supported financial services.

The seed component was designed to meet the expected need for certified cereal and potato seed that would allow farmers to achieve higher crop productivity in rehabilitated irrigated fields. The relevance of this component is called into question by the fact that effective demand for these seeds was low. This was found to be due to: (i) farmer uncertainty over the suitability of imported cereal varieties for local growing conditions; (ii) local consumers' taste; and, (iii) low levels of farmer incomes that have prohibited purchase of higher cost improved seed.

How effective were the implementing agencies in reaching their physical objectives?

The Project was very effective in achieving most of the physical targets set in the logical framework. As shown below, implementing agencies even surpassed irrigation, credit and seed component targets.

How efficient were the implementing agencies in achieving the maximum level of benefits with the resources allocated to them?

Beneficiaries rated project efficiency as high. Quantitative data also shows that the level of efficiency of the implementing agencies was quite high in terms of transforming financial inputs into physical outputs. This was especially true of WMS and IRP in irrigation, but also of ACBA in credit and to some extent ATG in seed supply. As noted above, funds made available in the Project were managed in such a way as to realize output levels that were higher than forecast at appraisal, in less time. The number of hectares covered in irrigation rehabilitation was, for example, 250% of what was planned. Likewise, the value of funds lent was 133% of planned levels. The Project was able to undertake more training days with allocated funds, establish more village associations for credit, and plant more hectares for seed multiplication.

However, these results need to be interpreted with caution. While at least partly attributable to good management, other factors came into play. They can be partly attributed to over-estimation of costs at appraisal, exchange rate fluctuations, and contributions in-kind from beneficiaries that exceeded expectations. In the case of lending, the good management that led to high repayment levels and the resultant possibility of increase the number of loans was the result of very conservative loan screening criteria. This had the undesired consequence of excluding beneficiaries the Project was designed to reach in the project area.

To what extent can the benefits of the various components be sustained after project completion?

The Evaluation found that the rapidity of project implementation which clearly contributed to its effectiveness and efficiency could potentially have a negative effect on the sustainability of its benefits: early project closure could cut short support for village level organizations created by the project. Many organizations, after only three years of project implementation, may not have had sufficient time to develop into mature institutions able to function on a sustained basis.

Irrigation Component Without a strong commitment from GOA to work with and support the WUAs, and translation of commitment into action, the benefits from irrigation rehabilitation investments will not be sustainable. Due to the dry seasons and problems in dealing with the water supply companies, WUAs currently face difficulties raising the funds from water charges to both pay for bulk water supply and fund O&M operations. Commercial farmers will be able to fund water charges. However, smaller non-commercial/subsistence farmers – potentially half the farmers in the project area - are unlikely to have acquired surpluses to pay the full costs. Inability to pay could well undermine the current model of irrigation water supply and management of irrigation O&M in Armenia.

Agricultural Credit Using Village Associations, ACBA has developed processes and procedures for managing a rural credit programme providing a range of credit from small seasonal loans of USD 100–200 through to longer term loans of up to USD 5 000. ACBA itself is providing agricultural loans for up to USD 40 000. Thanks to sound financial management and successful on-lending practices, there is every reason to expect that current levels of lending in Lori and Aragatzotn will continue. However, expansion in lending will be constrained by the ACBA levels of capital and access to additional credit lines. In future, savings mobilization from rural communities through the banking network created by the Project will be important for sustaining and expanding banking services.

Community Development Ongoing benefits from social development activities will depend on the energy and enthusiasm of the Civic Action Groups to mobilize resources for further projects to assist their communities. SHEN monitoring reports indicate that half of the groups created have taken steps to achieve this. Yet, without continuing interest and support from SHEN or a similar agency, this energy is likely to dissipate. The sustainability of the new economic development projects funded by loans from SHEN will depend on the entrepreneurship of the individuals who own the businesses that received them.

What were the costs and benefits of Direct Supervision of this project to IFAD?

Actual expenditures on IFAD direct supervision and follow-up of this project were approximately USD 65 000 per year. About USD 45 000 was covered by the IFAD regular budget and USD 20 000 came from extra-budgetary resources in the form of supplementary funds. The Country Portfolio Manager and Programme Assistant estimate that roughly 40% of their time was allocated to direct supervision of the project. Other IFAD colleagues indicated that this might be an underestimation of actual CPM and PA time dedicated. If this cost - normally covered in payments to Cooperating Institutions - is accounted for, then overall costs increase to USD 139 000. These costs, in terms of staff time, were borne by the CPM and Programme Assistant. They were incurred at the expense of other projects and IFAD duties that received less of their attention and at their own personal expense. The assignment of an Associate Professional Officer to assist the CPM added human resources (and costs), attenuating the burden on the CPM and PA for part of the project implementation period.

Comparison with other projects directly supervised by IFAD from among the current pilot programme and eventual "second generation" IFAD supervised projects will be needed in order to determine the extent to which staff time costs were high because of the first time or path-breaking nature of durect supervision in the NWASP case. However, while costs incurred in the recruitment of consultants for implementation support (in this case about two thirds of the USD 65 000) could be treated as variable, the basic cost to IFAD of direct supervision in terms of staff time is fixed. And, experience form this project would indicate that it is likely to remain higher than originally expected, even after experience is gained in the related processes. This will be due to the numerous time consuming tasks in loan administration, loan supervision and implementation support and because of the nature of the close working relationship that develops between the project management and the IFAD staff.

The most apparent benefits of direct supervision to the project were derived from the responsiveness of IFAD to project needs. This was manifest in rapid replies to queries from the PCU related interpretation of the Appraisal Report and IFAD procedures with respect to procurement, recruitment and other disbursement issues. It was a significant contributing factor to the quick pace of project implementation and full disbursement before the expected closing date.

Project implementing agency staff were generally appreciative of the quality of technical support provided by IFAD in project implementation. This is attributable, at least in part, to IFAD direct supervision as it heightened IFAD's understanding of project needs for support as well as its commitment to efficient provision of support. The project also appreciated the continuity of working with one interlocutor from design through implementation. Regrettably, the expected learning benefits to improve performance were not shared with IFAD generally, but rather confined to the staff directly concerned. Moreover, for the IFAD portfolio, the learning was limited as a result of the fact that many of the same individuals – both staff and consultants – were responsible for both project design and implementation. There was an advantage to this in that the individuals involved could learn, through direct observation, the outcomes resulting from their own decisions made in project design. That is, they could see for themselves what worked and what didn't. However, the main disadvantage was that no constructive critical review or questioning of decisions took place outside of a very small circle of individuals. Hence, there was a lost opportunity for them to learn from the perspectives of other professionals within IFAD who have different skills and experiences to draw on or who were not as closely caught up in the everyday project matters.

Back to Top

---

Project performance by component

Irrigation rehabilitation and water management. A total of 322 km of canals, pipes and associated control structures were installed bringing 19,688 ha of previously irrigated land back into full production. The Government adopted the WUA model developed by the Project for replication throughout Armenia. However, without the training courses and support services provided by NWASP, it has been considerably less successful outside the project area. Three federations/unions of WUAs have been formed in the project area to provide a link between users of common supply channels and to bring a united voice to work with government and water supply companies. Despite recent very low rainfall some systems were still able to achieve higher yields, even with limited water supplies. WUAs have had some success in collecting water charges where irrigation water has been available. However subsistence farmers may not be able to generate enough cash income to pay water charges which will compromise the whole WUA / O&M funding model. Communities are still confusing WUA charges with government tax collection.

Rural credit. The formal credit component of the Project has been implemented through ACBA using collateral-based lending. In 2000, 2 314 loans totaling USD 1 635 637 were disbursed to clients in project communities. ACBA successfully established branch offices in the three project marzer. IFAD loan funds provided to ACBA as a grant allowed it to increase its capital base and expand lending to more farmers and small business clients. While ACBA business suffered from serious mismanagement in Shirak, the contrary was true in Lori and Aragatsotn where loan repayments were close to 100%. This too allowed for expanded lending beyond levels forecast in the Appraisal Report. Village Associations were created in 186 project villages with 5 225 members to identify borrowers, screen applications and provide joint security for loans. Credit was short term and initially targeted at crop inputs, but is now being used to purchase large livestock. A significant weakness in design, related to the prevailing financial sector environment in Armenia, was the absence of a companion programme to mobilize savings in communities. In addition, ACBA proved unwilling to reach out to borrowers without collateral, despite IFAD Supervision Mission pressure to do so through group lending.

Community development. Villages in each marz based on poverty criteria were selected and provided with assistance to identify and undertake social and economic development projects. CAGs were encouraged to identify other social development projects that could be proposed to other organizations for funding. Sixty-one social development projects were completed. The majority were water supply projects, as these were achievable within the project-defined funding limit of USD 10 000. Half the CAGs have initiated further activities in their villages. At the request of communities, emphasis shifted from social development to group economic development projects, such as rehabilitation of machinery stations. It shifted again, in late 2000, to the provision of loans by SHEN to individual farmers to provide employment opportunities in their communities. Nineteen such activities were funded.

Seed production services. ATG efforts have been directed to increasing the supply of certified and graded cereal seed with some support to potato and alfalfa seed production. More than 900 t of elite and first generation seed has been produced on over 400 ha but has been limited by the dry seasons. Informal arrangements to distribute this seed to poorer farmers have been very successful using seed swapping and buybacks to overcome their inability to pay for the seed. ATG has also purchased large quantities of seed for distribution in other parts of Armenia to the benefit of USAID food security programmes. Yet, the seed production programme has been clouded by disagreements over consumer acceptance of US wheat varieties and the need to use higher seeding rates for Armenian and Russian seeds. Performance is declining in the seed and input credit programmes operated by ATG. Seed testing equipment provided to MOFA cannot be fully utilised because funds were not allocated to training.

Animal health. Inputs were financed by the IFAD loan to upgrade the marz veterinary laboratories. These have not been accompanied by the government regulation required to support private veterinarians. Therefore the project could not provide the planned equipment and training to the private sector. Nonetheless, private veterinarians are ensuring essential vaccination and animal health programmes in the project area.

Project coordination. The performance of the PCU and project coordination in general has been exceptionally good. Excessive audit and checking on PCU activities by various government authorities had high costs in terms of staff time and would appear unjustifiable. Good management combined with the use of the SOF provided before the loan signature and good support from the IFAD Country Portfolio Manager (CPM) led to not only meeting but also exceeding targets in virtually all components, even ahead of schedule. The Project Coordinating Council (PCC) met regularly. However, project design did not include a formal coordination or planning group at the level of individual marzer, which would have permitted better decentralized bottom-up planning. In implementation, the inputs of the marz M&E coordinators, who all had very close relationships with the marz governments compensated for this.

Back to Top

---

Project impact

Each of the project components led to successful outcomes. As noted above in paragraphs 13.-15., there have been improvements for beneficiaries in terms of production and food security, even in the face of declining per capita income levels for beneficiary and control groups. However, it is very difficult to draw definitive conclusions about the project impact at the household level due to: (i) the short time that has elapsed between surveys; (ii) differences in timing, within the agricultural calendar, of the surveys (August 1998, October 1999 and March 2001); and, (iii) incidence of drought between the first repeater survey and the survey done for the Evaluation.

More than 36 000 households directly benefited from some or all project activities - roughly 250% of the 9 500 households the project was expected to reach. More than 20 000 families have benefited from the irrigation rehabilitation activities servicing 18 700 ha. A further 18 000 benefited from the 78 social and economic development projects facilitated by SHEN. ACBA credit activities included 7 466 loans. In the year 2000, for example, it reached 2 400 households with credit for farming crop inputs and livestock to improve their businesses and take better advantage of upgraded irrigation facilities.

Apart from household level impact with respect to food security and production levels, project impact on households and communities is detectable, in terms of social, economic and technical practices. Cropping patterns, cropping practices, areas planted and yields have changed because of the project, for example, cultivation is intensifying and land parcels are becoming larger. Economic activities, such as marketing have changed - a lower proportion of food crops are being marketed and credit is being used to purchase inputs. Social behavioral patterns are changing, as a result of the project people are forging new village-level working relationships in community organisations that replace old politically oriented ones. However, the project has been operating for only three years and it is one of many new forces of change in the project area, not all changes observed can be fully attributable to the project.

Based on Evaluation observations the greatest long term impact on the communities served by NWASP are likely to come from the development and support inputs provided to grass roots organizations. The 186 ACBA village associations, 58 water user associations, and 55 civic action groups created in the project area have begun to develop the capacity to sustain current benefit levels and undertake further activities. However, they have not yet had time to take root and are not fully prepared to assume their responsibilities for physical outputs. From this point of view, project phasing was sub-optimal and future benefit streams will be in jeopardy if the project is closed prematurely because funds are fully disbursed.

To consider the beneficiary point of view, the Evaluation team compiled responses obtained at the village level whereby the beneficiaries rated the impact of project activities by component, based on their own subjective definitions of impact. These ratings, shown below, were combined with beneficiary ratings of project according to criteria set by the CLP and specified in the key questions of the evaluation.

Evaluation Teams Synthesis of 32 Community Assessments of Project Activities

---

Recommendations

Project closure

• The impact and effectiveness of NWASP activities should be monitored up to the original planned completion date of December 2002. This should focus on the performance and sustainability of the grassroots organisations created through NWASP.

Irrigation

• Fully transparent information on the ways that water fees are set, the ways that fees collected are allocated among concerned agencies, and the ways that fees are used by those agencies should be made available to all water users through water user associations.
• An independent study of the cost of water delivery should be immediately undertaken.
• On the basis of the results of study there should be a: (i) review of government policies on the allocation of investment resources to irrigation infrastructure; and, (ii) re-examination of the viability of subsistence and small-scale farms given the costs of water that they may have to bear.
• Current policies regarding use of loan funds for civil works should be changed to allow water user associations to supervise and undertake rehabilitation of schemes with technical backstopping by concerned government agency and monitoring by the project.

Credit and finance

• New approaches to providing financial services to borrowers without collateral should be tested.
• Involvement of non-financial institutions should be limited to: (i) support borrowers in accessing and using finance; and, (ii) support banks in loan screening, disbursement and repayment monitoring.
• Financial services in rural areas should be expanded to include savings and establish links between savings and lending practices.

Seeds

• Testing of new cereal varieties should be done by GOA research stations, with links to producers.
• Grassroots seed producer organisations, based on the models of Vardablur and Gulagarak, in Lori should be initiated at district level in each marz to allow growers to share technical and market information.

Community development

• Community development agencies should not be directly involved in lending activities as such, especially for loan-sizes already handled by existing financial institutions.
• Community development agency involvement in financial service delivery should be limited to support microfinance activities not normally handled by banks.
• The approach to the use of grant financing to reduce poverty through community development should be broadened. If communities wish to focus on employment generation, for example, a range of activities such as training and assistance to entrepreneurs in preparing business plans, accessing formal institutions, or investing remittances could be considered.
• The NWASP model of developing and supporting grassroots organisations such as the community action groups, water user associations and village associations, to assess people's priorities, plan, and implement activities should be used wherever possible.
• Assigning local NGOs the responsibility for working with projects to develop these grassroots organisations should be considered.

Information

• All projects should include communications programmes to provide target groups with information they need to understand project objectives and activities. Widespread information campaigns should start during project design and continue throughout implementation.

IFAD supervision of IFAD loans

• There should be a monitoring system to analyse financial and human resource costs to IFAD of direct supervision.
• There should be explicit ways of sharing, within IFAD, the knowledge that it acquires through direct supervision.
• A system of checks and balances should be created whereby responsibility for design, supervision, monitoring, and approval processes are shared rather than vested in a single staff member as at present.

---