Completion Evaluation

Introduction

Evaluation objectives and procedure. The completion evaluation of the Niger Special Country Programme – Phase II (PSN-II) was conducted by the IFAD Office of Evaluation two years after the programme's closing. It aimed, in the first place, at ensuring accountability of IFAD and its partners on results obtained by co-financed operations and, in the second place, to contribute to learning by project partners, in order to improve implementation of ongoing projects and programmes and the design of future interventions in the Niger and the region. Three aspects have been evaluated: (i) programme performance (relevance, effectiveness and efficiency); (ii) programme impact on rural poverty in various domains as well as sustainability, innovations and up-scaling; and (iii) the performance of IFAD and its partners. Following a preparatory mission in July 2006 and preliminary work to gather information at the programme sites, the main evaluation mission took place from 19 September to 17 October 2006. Three sources of information were used: interviews with programme participants, partners, former personnel and beneficiaries; a bibliographic study; and observations in the field. At the mission's end, a meeting was held at the Ministry of Agricultural Development and an aide-mémoire was presented to the Ministers of Agricultural Development and Livestock Resources, as well as to the Commissioner for Rural Development of the Ministry of Finance and Economy.

Country context. Niger is one of the world's poorest countries. Although agriculture employs some 87 per cent of the population and contributes 40 per cent of GNP, it is highly vulnerable to natural phenomena. Much of the country is desert. Arable land and pastoral lands cover just 3.5 per cent and 9 per cent of its territory respectively, concentrated in the Sahelian southern strip. Niger is regularly exposed to food insecurity and must resort to imports and international aid, as shown in the most recent crisis in 2005. Approximately two thirds of the population live below the poverty line, and a full one third below the extreme poverty line. This situation is attributable to a number of factors: climate constraints, sandy soil with limited potential, strong demographic pressures leading to imbalanced resource use, limited access to inputs and equipment that would favour intensification, and serious inadequacies in basic services and infrastructure. Women and pastoralists are particularly vulnerable, although the 2005 crisis pointed out high poverty levels in dense agricultural areas as well, among farmers having lost their land.

IFAD strategy in the Niger. The 2006 Country Strategic Opportunities Paper (COSOP) calls for reducing rural poverty in accordance with the Poverty Reduction Strategy Paper (PRSP) under three major thrusts: (1) reduce vulnerability and help rural households achieve food security; (2) improve incomes and market access for target groups; and (3) improve basic social services for vulnerable populations.

The project. PSN-II covers a large part of the Niger's territory, in four major agro-ecological areas: the agricultural area irrigated by the Niger River to the west, the agricultural area irrigated by the Komadougou to the east, the rainfed farming area to the west and centre (Ouallam, Loga, Illéla), and pastoral lands (Tchintabaraden, Abalak and Tchirozérine). The target group includes 900 irrigators (300 of them women), 74 000 farmers in 130 village terroirs¹

(260 villages) and 30 000 pastoralists in 40 pastoral terroirs, representing 4 000 families.

Project objectives. The overall objective of PSN-II was to sustainably restore conditions for balanced development based on the optimal use of all natural resources. Specific objectives were: (i) to contribute to achieving food and income security by increasing agricultural production; (ii) to help restore and maintain the productive potential of agricultural and pastoral ecosystems through promotion of soil and water conservation and agroforestry activities; (iii) to help establish conditions for self-managed socio-economic development by promoting farmers' and pastoralists' organizations; (iv) to encourage partnership between community-based organizations and the private sector, and participation by women and youths in the decision-making process of their communities; and (v) to assist in laying the ground for self-sustaining development through the promotion of mutual savings and credit funds, in partnership with the formal banking system.

Programme components. These correspond to the four operational objectives: (a) increasing agricultural production and ensuring sustainable production systems in small-scale irrigation areas (Tillabéri and Diffa); (b) improving food security and helping to increase the incomes of small-scale producers through a terroir management approach (Illéla, Loga, Ouallam and Diffa); (c) improving living conditions for populations in pastoral and agro-pastoral areas (Tchintabaraden, Abalak and Tchirozérine); and (d) enabling women, youths and small-scale producers, once trained, to access working credit to raise their living standards. In addition to these four components, there is an institutional support and coordination component.

Project performance

Design. The component objectives are in line with Niger's policies (terroir management, professionalization of microfinance and irrigation development) and consistent with IFAD's country strategy. However, the programme's national and multisectoral ambitions, little discussed at the design stage, led to two major constraints on PSN-II's effectiveness and efficiency: the project's geographical dispersion as a result of circumstances in the wake of the crisis of the 1990s which favoured a comprehensive programme on a national scale, and dispersion of intervention themes encompassing very different issues and challenges: irrigated farming, rainfed farming and pastoral areas. Given the programme's ambitions and structural weaknesses within Niger's state apparatus, the operating resources provided for in design were insufficient. Also, the lack of a logical framework from the outset was a major handicap to coordination, understanding, steering and monitoring and evaluation.

Partnerships. With faire-faire (outsourcing) as the underlying principle, the project operated mainly through contracting and supervision of recognized service providers. Some activities had to be delegated to various public crop, livestock, rural engineering and agricultural research departments. But the programme design provided for important human resources and management capacity to be maintained within the field units. There was no effective delegation of responsibility to operators, and little visible complementarity between field units and operators in operational terms. Given such multiple participants in diverse contexts with heterogeneous capacity and experience, truly exceptional coordination, training and supervision was needed in the programme management unit – an unrealistic expectation in view of the country context at the time.

Funding. As of end-2004, IFAD loan disbursements stood at SDR 7.95 million (381-NE) and SDR 1.6 million (SRS 46-NE), resulting in disbursement rates of 99.02 per cent and 23.7 per cent. For PSN-II as a whole, funding reached 86.4 per cent of projections at the time of the programme's closing. At the Government's request, Annex 2 to Loan Agreement No. 381-NE was amended twice (November 2001 and August 2003) to reallocate expense categories. As a result of its high geographic dispersion and cumbersome coordination arrangements, the programme far exceeded the expenditures provided for in the loan agreements with respect to project operation, technical assistance and per diem allowances. Spending for rural finance, training, land reclamation and soil conservation works as well as rural infrastructure remained well below projections, although some ground was gained following the mid-term review in connection with physical investments.

Programme coordination and management through the programme management unit and seven field units performed poorly because of external constraints (suspension of the loans, communication problems owing to dispersion, cumbersome arrangements, etc.) as well as internal ones (lack of a logical framework and a functional monitoring and evaluation system, dissension among managers, weak administrative personnel, limited skills development for personnel and contractual operators, centralization of contract management, etc.). Although corrective measures were proposed by successive supervision missions, it was only after the Mid-term Review – which took place late – that the programme was restructured in early 2003, in order to lower operating costs.

Implementation of the small-scale irrigation component. A participatory agricultural appraisal took place first of all to determine constraints and guidelines for training and research by perimeter. Rehabilitation studies were conducted on the 28 sites selected. Rehabilitation work was undertaken on 20 collective irrigated perimeters (CIPs) but greatly delayed by slow administrative procedures and financing delays. In the Komadougou river valley, unresolved land disputes prevented rehabilitation work from commencing. Motorized pumps were repaired for one third of the CIPs. Agricultural extension activities on 15 useful technical topics reached 300 farmers in Diffa and 500 to 600 in the Niger river valley. Promotion activities for producers' organizations reached a larger number of groups than planned. Finally, promising agronomic research took place on a small scale in partnership with the National Institute for Agricultural Research (INRAN), based on tests and evaluations with producers, but little effort was made to disseminate the results.

Implementation of terroir management components in farming and pastoral areas. An investment was made at the outset in designing terroir management methodology. The sociological land-based approach proposed by international technical assistance was appreciated by staff for its modernity but was soon abandoned as overly cumbersome. Much work was done on information and terroir selection, particularly in the pastoral area, prior to undertaking participatory appraisals and local planning. In farming areas, 60 terroir development plans were prepared out of 170 planned, while in the pastoral area only 10 such plans were produced, out of 40 planned. Fewer than half the plans were implemented, for financial and organizational reasons. Among physical accomplishments, soil and water conservation and land reclamation were most significant, while infrastructure remained well below projections. Training reached a total of 21 000 people, or 4 per cent of the target.

Implementation of the rural finance systems component. Three specialized credit facilities were put in place with different objectives, principles and operators for each area. Under facility I, whereby SONIBANK credits were intermediated by a local NGO in the river area and farming terroirs, close to CFAF253 million in credits were granted, of which 85 per cent as short-term credit (4 907 beneficiaries) and the remainder as medium-term credit (355 beneficiaries). The volume of credit granted was lower than expected, mainly because of delays in guarantee fund availability and overly prudent behaviour on the part of SONIBANK. Under Facility II, whereby credit was provided through a local mutual savings and credit fund in the Diffa area, CFAF83 million in credits were granted, of which 92 per cent as short-term credit (711 beneficiaries) and the remainder as medium-term credit (81 beneficiaries). Slow decision-making processes were a constraint. In the pastoral area, where facility III was to promote the creation of local savings and credit funds, small credits (CFAF5 000 to 40 000) on very short terms (one to two months) were made to just four such funds. In the absence of rigorous monitoring, the total volume of credit granted by these local funds is not known, but CFAF3,38 million was available to them, including CFAF0,48 in the form of savings by their members. This very low level is attributable mainly to the fact that guarantee funds were made available late and were insufficient.

Achievement of outcomes. The project's management problems had very serious consequences (an insurmountable delay in investments, a loss of trust by partners and the demobilization of equipment), and a considerable adverse effect on the achievement of objectives. Expectations for adapting the terroir management methodology were met only in part. The approach developed raises many questions: cumbersome processes, weak project feasibility, slow decision-making for implementation, little accountability by local authorities. Despite poor capitalization of research, tests in the irrigated area did yield some results for onion, pepper and tobacco crops. There is no question that PSN-II met expectations best with respect to agricultural extension in the irrigated farming areas, where adoption rates reached 80 per cent for more than half the techniques taught. Nevertheless, farmer support neglected several factors with a considerable impact on the development of land for irrigation. These include household production systems, market access and secure land tenure for farmers. Building collective capacity remained limited in the absence of a longer-term vision and consideration of key issues in sub sector development and legal recognition. Soil and water conservation and land reclamation activities clearly enabled arable and sylvo-pastoral land to be recovered. The fattening of animals for market, considered a profitable activity, was assigned priority for credit under facility I, but repayments were modest since there was little follow-up after the service provider contract ended (92 per cent for short-term credit and 88 per cent for medium-term credit). As to credit under facility II, 83 per cent was provided to purchase inputs, and the mutual fund in the Diffa area successfully recovered the full amount through rigorous and regular follow-up.

Summary: programme relevance, effectiveness and efficiency. PSN-II's major orientations and design principles were relevant, but its proposals often lacked realism and operationality, particularly with respect to context and actual capacity. Serious management and organizational problems, in combination with geographically and thematically disperse arrangements, led inevitably to a low level of effectiveness, with only a small part of objectives actually achieved. Despite the reorientation that took place following the mid-term review at the end of 2002, operating costs at 230 per cent of projections with investments of less than 50 per cent in most cases clearly illustrate PSN-II's limited effectiveness.

Performance by partners

Performance by IFAD. Upon completion of the evaluation, IFAD's performance appears moderately unsatisfactory. Although IFAD made an effort to resolve the critical situations encountered, it was not successful in negotiating a realistic programme in terms of implementation conditions, and its contribution to support, monitoring and supervision was too limited given the particular programme and country challenges. The Mid-term review took place too late and the Project Completion report adds little to the Government's one. IFAD did however draw important strategic lessons for its future programme in the Niger.

Performance by the Government of Niger appears moderately unsatisfactory as well. Even taking into account the very difficult socio-economic and political conditions prevailing during the PSN-II implementation period, the loan repayment defaults and lack of decision on project management arrangements led to many problems. The Government lead supervision missions were rigorous and effective but too infrequent.

Performance by UNOPS is considered moderately satisfactory. Synergies were lacking with the programme management unit and partners, that might have avoided the suspension of land reclamation activities for several six-month periods. Although the missions conformed to the mandate, they lacked adequate resources in proportion to the programme's scale and diversity, and were too infrequent and too short to come up with recommendations beyond simply general management issues. The problems caused by the change in officers responsible for supervision and the mismatch between the goal of building capacity among participants and centralized decision-making procedures were noted as well.

Performance of service providers. Performance of NGOs under contract for implementing the terroir management approach and of service providers for rural credit and in the irrigated areas is considered moderately satisfactory. They were able to mobilize quality personnel and carry out their mission relatively well, despite the breaks in cash flow and delays in the provision of funding. They have shown that they were able to capitalize on the PSN-II experience.

Programme impact

Overall impact. Compared to the programme's aspirations and the substantial human and financial resources used, in particular the US$20 million in loans, PSN-II had a limited impact. Although this was one of the country's largest projects, its impact was clearly diminished by the serious problems encountered: a low level of effectiveness (breaks in financing and operating problems), problems with design quality and implementation in many instances, the lack of a sound strategy to reduce vulnerability among target groups, and an insensitive approach to providing support in the absence of a strategy to empower the organizations concerned.

Targeting of measures to reduce poverty and inequality. The proposed targeting strategy was justified in terms of choice of intervention areas. However, it was not sufficiently well-defined to bring about activities benefiting the most disadvantaged and none of the components proposed a targeting method. In the absence of a sound strategic and methodological approach to reduce vulnerability among the poorest, terroir management activities were too superficial to have an effect on highly disadvantaged groups. The same is true of small-scale irrigation. Nor did the credit facilities meet the needs of the most disadvantaged, including women and young people.

Environment and common resource base. PSN-II has improved access to irrigation water for farmers on perimeters. On plots having benefited from soil and water conservation and land reclamation measures, the impact is apparent in biodiversity, reforestation, erosion control and the expansion of treatment. It is highly regrettable that the impact on communal resource management systems is negligible. The programme's impact is considered moderately unsatisfactory overall in terms of the environment and the management of natural resources.

Agricultural productivity. Here the impact is greater thanks to agricultural extension and soil and water conservation combined with land reclamation measures. Farming on CIPs has increased greatly as a result, although it is gradually being abandoned because of land tenure insecurity and lack of capacity to repair motorized pumps. Introducing erosion control measures has increased crop yields by 30 per cent for farmers with land.

Food security. The programme's impact in this regard is considered moderately unsatisfactory. Irrigated crops are self-consumed in part, but the insecurity of land tenure makes them an unreliable source. In Diffa, CIPs are not deemed to contribute much more to producers with plots outside the collective perimeter. Still, soil and water conservation and land reclamation efforts have had a decisive impact on the agricultural development of land. Cereal availability has increased for those households with land, although the cereal banks are mainly ineffective. Returns on microcredit have generally been reinvested in household nutrition.

Social capital and empowerment. Impact in this area is judged moderately unsatisfactory, despite the concern to involve women and young people. Skills among water user groups remain weak in infrastructure management, maintenance and repair. Access to inputs and marketing was not mastered during the programme. In pastoral areas social networks around the investments were strengthened by consultations and exchanges in connection with PSN-II. A change in behaviour favouring women was noted as a result of the creation of savings and credit funds in these areas.

Financial assets. Generally speaking, insufficient institutional support was provided to build capacity among credit operators that would have enabled them to pursue lending activities with poor and vulnerable populations in the programme area.

Sustainability and ownership. Overall, these are unsatisfactory in the absence of any strategy to ensure the permanence of the organizations and services set up, in irrigation as well as terroir management and micro credit. The project was unable to take advantage of opportunities to transfer their functions and capacity to private or public operators or to involve permanent organizations (communes, land tenure committees, farmers' organizations) to follow up on the organizations created.

Innovation and up-scaling. Contrary to expectation, PSN-II was not very innovative. Promoting and disseminating local innovation was never a concern for project participants. Nor were the few programme innovations capitalized upon or disseminated.

Conclusions and recommendations

Overall assessment. PSN-II was an ambitious programme. All the partners involved had high hopes, and today there is a sense of great disappointment and bitterness among them. IFAD has however drawn important lessons from the experience, as reflected in the new COSOP and the IRDAR project. The overall assessment of PSN-II is moderately unsatisfactory. A comparison of PSN-II's scores with average scores for IFAD-financed projects evaluated in 2005 shows similar trends (relevance being the most satisfactory and sustainability the least), but overall PSN-II is assessed one or two points below the average. Particularly weak compared to the 2005 average are effectiveness, efficiency and innovation. The table hereunder presents ratings given by the evaluation to the project for the different evaluation criteria.

Performance ratings ² for the different evaluation criteria

Ratings range from 1 = highly unsatisfactory to 6 = highly satisfactory        
Source: PSN-II Completion Evaluation

Recommendations. Three types of recommendations are proposed, based on lessons learned from the PSN-II experience. Two strategic recommendations relate to targeting and project feasibility, and capitalizing on experiences. Three recommendations relate to operational issues, i.e. essential conditions for effective outsourcing, ways of ensuring true accountability and capacity-building in project management and supervision. The last three recommendations have to do with three thematic issues that remain current in the Niger: the challenges of developing small-scale irrigation, key thrusts in developing pastoral areas, and building capacity among microfinance institutions.

---

1/ There is no precise translation for the French word terroir, which indicates a socio-culturally, geographically limited land area valorised by a relatively stable community. It is often easier do identify the terroir of a settled community compared to the sometimes very vast terroirs of nomad or semi-nomad groups. Several communities may charge parts of a terroir with each other during certain periods of the year.

2/The rating scale ranges from 1 to 6 as follows: highly successful (6), successful (5), moderately successful (4), moderately unsuccessful (3), unsuccessful (2), highly unsuccessful (1).