Agreement at completion point - Completion evaluation

Evaluation objectives and process. In June 2008, the Office of Evaluation (OE) of IFAD conducted a Completion Evaluation of PHBM II in Madagascar. The main objectives of the evaluation were to assess the performance and impact of the project; and generate lessons and recommendations for the preparation and implementation of other rural development interventions in the country. The evaluation was conducted following the most recent OE project evaluation methodology, and focused on five dimensions: (i) project performance in terms of relevance, effectiveness and efficiency; (ii) project impact on rural poverty; (iii) sustainability of achievements; (iv) innovations, their replication and scaling up; and (v) performance of partners, including IFAD and the Government of Madagascar. A preparatory mission was fielded in May 2008, followed by the main evaluation mission from 9 June to 4 July 2008. On 2 July, the aide-mémoire of the main mission was presented to regional project partners at Fort Dauphin and, on 4 July, to the Ministry of Agriculture, Livestock and Fisheries in Antananarivo. In November of that year, the evaluation report was shared with the Government and other evaluation partners for comments. In order to generate inputs by project partners for the preparation of the Agreement at Completion Point (ACP) of the evaluation, a half-day workshop was organized in Fort-Dauphin on 3 December 2008, in combination with the project's completion workshop.

Core learning partnership. While stressing the need for independent evaluation, IFAD's Evaluation Policy underlines the importance of the participation of key partners throughout the evaluation process. Such participation is essential, as it engages partners in fruitful collaboration, enhances learning and facilitates the operationalization and adoption of evaluation recommendations. To that end, a Core Learning Partnership (CLP) consisting of the main users of the evaluation was set up, comprising  representatives of the Government of Madagascar (Ministry of Agriculture, Livestock and Fisheries (MALF)); (2) the project office; (3) IFAD (Division for Eastern and Southern Africa and OE); and (4) the United Nations Office for Project Services (UNOPS). Other project and IFAD partners in Madagascar were consulted during the evaluation, including: regional authorities and technical services; communes; main project service providers (AHM-KIOMBA, CALL, ASOS, ICAR, IFRA, etc.); the Mutual Savings and Loan Institution of Mandraré; the Centre for Agricultural Services (CSA) of Tsivory; officials of the Transport Sector Programme (TSP) and the Development Intervention Fund (DIF)¹; the National Water Authority; the Road Authority; the European Delegation;  the Ministry of Energy and Mines; and the Agricultural Research Centre (FOFIFA).

Agreement at completion point. At the end of each independent evaluation, an ACP is prepared by OE and the IFAD division involved in the assessment as well as the Government concerned. This action-oriented document shows how partners have understood the evaluation's findings and recommendations, and sets out the modalities for implementing such recommendations. Thus, the ACP aims at: (i) clarifying, deepening and rendering more operational the recommendations of the evaluation, and indicating to what extent and why they were retained by IFAD and the Government; and (ii) setting out findings and lessons calling for further discussion and reflection.

KEY EVALUATION FINDINGS

Genesis, institutional framework and financing of the project. The first phase of the Upper Mandraré Basin Development Project (PHBM I) (1996-2000) was identified following the 1991 food crisis that severely affected the south of the country after several years of drought. Targeting four communes (covering 3 233 km2) for a total of 39 800 inhabitants, the project was evaluated by OE in 2000. The IFAD loan amounted to US$6.9 million.

The second phase, PHBM II (2001-2008), was designed during 2000 at an estimated cost of US$23 million. The goal of the project was to help reduce rural poverty by: (i) diversifying and increasing rural incomes; (ii) improving food security for rural households; and (iii) contributing to the restoration and sustainable management of natural resources. The highly concessional IFAD loan of US$12.6 million accounted for approximately 55 per cent of the total project cost. The remainder was provided by contributions from the Government (US$5.2 million) and beneficiaries (US$1.8 million) and by cofinancing from two World Bank projects (PST and FID) for a total of US$3.6 million. The project was structured around five components: strengthening local capacity (7.7 per cent of total project costs); support for local initiatives (46.6 per cent); support to financial services (4.6 per cent); rural roads (28.2 per cent); and coordination, management and monitoring and evaluation (M&E) (12.9 per cent). PHBM II targeted the entire 96 000 population of the project area (17 400 households) and covered nine rural communes (over 8 600 km²), which increased to 11 in 2003 as a result of administrative reform and included the four communes covered by the first phase. It was implemented by the MALF and supervised by UNOPS, which was IFAD's cooperating institution until 2007.

Main outputs. As a whole, project performance was good in terms of physical and financial outputs, reaching a physical implementation level of the order of 120 per cent and a disbursement rate of almost 100 per cent. In terms of local capacity strengthening, the project supported the preparation of 11 municipal development plans and 135 village community development plans, and contributed to the organization and capacity-building of almost 1 500 farmer organizations (water users' associations, infrastructure management committees, informal production groups, etc.). The project established and trained development facilitators at the commune and community levels; supported literacy training for more than 6 600 beneficiaries (50 per cent of whom were women); conducted information, education and communication campaigns on nutrition and community health; and promoted rural communication through a rural radio. The project financed some 1 130 mini projects in the fields of agriculture, livestock and the environment through a local initiatives support fund, for the purpose of strengthening agricultural and livestock infrastructure (irrigation infrastructure on 2 250 ha, more than 6 000 subsidies for agricultural equipment, 46 passageways for vaccination of cattle, etc.); and developing economic and social infrastructure (15 km of community tracks, 21 small road works, 36 water points, nine schools and five health centres). The project rehabilitated 140 km of provincial roads, of which 79 km in partnership with the TSP, and 64 km of inter-communal tracks. In terms of support to financial services, the project set up a proximity financial services network, including a mutual savings and loan institution and nine branches at the commune level for a total of about 3,500 members.

Project performance. As far as project relevance is concerned, the objectives of PHBM II corresponded well to the needs of the rural poor in the intervention areas and were well aligned with Government and IFAD policies and strategies. The project followed an integrated development strategy that combined the participatory planning and management of local development, a consistent technical approach and an economical approach that focused on cofinancing mini projects, promoting promising value chains and developing proximity rural microfinance services. Project design made provision for adequate resources (human, financial and technical) to successfully support the local initiatives of rural populations. However, it also contained a number of weaknesses, in particular: the small size of the social infrastructure subcomponent (health, education and potable water supply) and absence of strong partnerships to complement investments therein; and the fact that no consideration was given to the possibility of public technical services acting as strategic partners in project implementation. However, at the request of the Government and with support from IFAD and UNOPS, adjustments were made to the design during implementation with a view to improving project performance. Examples here include the mobilization of several partners not foreseen in the original design documents, introduction of the commune level as the centre for local planning, and application of the value-chain approach. In all, the evaluation considered the project's relevance to be moderately satisfactory (score of 4)². 

Project effectiveness was, overall, satisfactory (score of 5), albeit somewhat variable depending on the specific objective (SO) considered. As shown above, PHBM II was effective in: strengthening local capacity to plan and manage economic and social development (SO1); supporting local initiatives that contributed to higher and more secure incomes, greater food security and more sustainable management of natural resources (SO2); and better use of production by opening up the project area, improving access to markets, trade information and financial services, and reducing post-harvest losses by supporting the processing and marketing of produce (SO4). Performance was only moderately satisfactory in terms of improving access to financial services for the rural poor, especially women, youth and the landless (SO5), mainly owing to poor mobilization of savings and insufficient coverage of villages outside the main towns. In terms of promoting the development of sustainable agricultural production systems to restore and ensure the sustainable management of natural resources (OS3), the project's effectiveness was moderately unsatisfactory, especially because activities were mostly limited to rather scattered demonstrations. Moreover, the project lacked both a vision for and approach to sustainable management of natural resources (water and soil), especially for irrigated land. 

The project demonstrated a good overall level of efficiency (score of 5), having reached an estimated 30 percent economic rate of return for productive investments (irrigation infrastructure and mini-projects for agriculture, animal husbandry and reforestation) thereby making it possible to recover the amounts invested even before the project came to an end. In contrast, however, the efficiency of the rural financial services component was relatively poor owing to constraints having to do with the project area (isolation, low population density) and limited technical and managerial capacity of staff and advisers.

Rural poverty impact. The project had a clear-cut impact on household income (score of 6), thanks to an average increase of almost 75 per cent in cash income per capita between 2002 and 2005. This was accompanied by successful targeting of the most vulnerable farmers, who saw their income increase by 160-180 per cent. There has been a considerable increase in agricultural production (score of 6) - close to 90 per cent for rice and 120 per cent for cassava - and food security has improved greatly, with the proportion of food self-sufficient households rising from 36 per cent in 2001 to 78 per cent in 2008. This was possible thanks to the application of a value-chain approach focusing on three commodities (irrigated rice, garlic and onion) benefiting from a surge in demand from the rapidly developing city of Fort Dauphin and the global price increase of cereals. Development of garlic and onion production in the project area was driven mostly by women. However, the reach of this impact is rather limited because only about half the households in the project area produce these commodities. Furthermore, the involvement of young people in project interventions was very limited, which meant that they benefited only indirectly. As for social capital and empowerment (score of 5), the project was instrumental in launching a radical transformation of the social environment through the promotion of a new culture of dialogue at all levels and by organizing the population, thereby providing real opportunities to communities to participate in the planning and management of local development, and with genuine bargaining power vis-à-vis various local stakeholders. However, project impact was less evident in the field of natural resources and the environment (score of 3). Little progress was noted in the management of productive assets at the farm and community levels (inputs, water, soil and fertility, labour force, secondary crops, livestock, etc.). As far as institutions and services (score of 5) are concerned, project interventions led to a radical institutional transformation thanks to capacity-building and revitalization of communal structures (communal development committee, communal council) and the establishment of a Centre for Agricultural Services (CAS), a network of proximity microfinance services and an inter-communal land tenure office. By contrast, the project's contribution to consolidating and revitalizing decentralized public technical services was limited. On the whole, the overall impact of the project is considered satisfactory (score of 5).

Sustainability. Several factors contribute to the sustainability of PHBM II achievements, such as the continuing economic potential of the value chains that benefited a large proportion of the population; simplicity of the design and operation of irrigation systems; emergence of a new awareness among the population as to the need to protect natural resources, etc. However, there are also a number of serious risks to the sustainability of achievements, such as the fragility of, and lack of maintenance on, the rural road network that still has some black spots; and the poor capacity of local public technical services and newly- created institutions (CAS and inter-communal land tenure office); inadequate health, education and drinking water infrastructure; and lack of a vision for and approach to the sustainable management of natural resources. These risks are mainly the result of weak project design (lack of strong partnerships, and inadequate involvement of public technical services). Therefore, the evaluation concluded that project sustainability should be considered as moderately unsatisfactory (score of 3).

Performance of partners. On the whole, the performance of the Government and its various agencies has been satisfactory (score of 5) inasmuch as they steered this relatively isolated and complex project in an adequate fashion while granting extensive autonomy to the project office. The communes have been actively and wholeheartedly involved in project implementation. The project management unit was particularly successful in mobilizing a large number of partners that were not originally foreseen, in taking many good initiatives for increasing the project's relevance and effectiveness, and ensuring proper planning and coordination of activities. The M&E system, despite its delayed start-up, is exemplary. IFAD's performance is considered only moderately satisfactory (score of 4) owing to (i) some weakness in project design, indicating a lack of consideration for a number of important recommendations by the Interim Evaluation (strengthening of the education, health and drinking water components; specific targeting of youth, in particular through vocational training; and reducing disparities between irrigated and rainfed areas); (ii) the absence of well-established partnerships; and (iii) the focus of the participatory approach solely at the village level. IFAD has nevertheless demonstrated flexibility and support for needed adjustments to design, which enabled the project to achieve a good level of implementation performance. UNOPS has shown satisfactory performance (score of 5) by conducting regular supervision missions, always with the same teams, which contributed to the effectiveness of its services. In addition, the supervision findings and recommendations have often exceeded the status of technical and financial project implementation to cover deeper technical, methodological and economic aspects that contributed to improving project effectiveness.

In conclusion, the overall assessment of PHBM II is satisfactory (score of 5). Project performance was good, largely thanks to its integrated local development approach, an excellent project team performance and consistent support to project implementation on the part of the Government, UNOPS and IFAD. Project impact on rural poverty was significant, particularly in the areas of food security, incomes and social capital. However, the evaluation finds that there are still a number of significant risks that endanger the sustainability of project achievements.

Key recommendations of the evaluation approved by the partners

The Completion Evaluation of PHBM II makes four recommendations. The first three relate to important strategic lessons that should be taken into account for any future rural development projects in Madagascar. The fourth recommendation concerns the consolidation of project achievements in the area of intervention because the sustainability of project impact remains at risk.

Recommendation No. 1 (Government, technical and financial partners): To promote the balanced and sustainable development of rural areas of Madagascar, by better harmonizing interventions of integrated participatory local development projects with national, sectoral and regional development programmes, on the basis of clear and firm partnership commitments.

Local development projects have the advantage of developing considerable knowledge of the environment and sound expertise in the management of local development. Such projects support local initiatives and are planned and implemented in a participatory manner, thus ensuring proper consideration of people's needs and good ownership of project results. However, local development projects often have limited resources compared to needs, as was the case for PHBM II. In contrast, while sectoral programmes (agriculture, health, education, transport, etc.) are generally equipped with significant financial and technical resources, they lack the local anchorage required for sustainable development to meets local needs. Thus, the combination of a local development project, integrated with prior or simultaneous sectoral programmes, appears to be the winning formula for generating a better harmonized economic and social development, and a more consistent and durable impact.

To ensure proper coordination and synchronization between local development project interventions and sectoral support programmes, it will be essential to forge strong, formal partnerships based on the complementarities of both types of projects. It will be necessary to go beyond making an inventory of what sectoral programmes intend to do in the project intervention area, and to ensure that any agreements negotiated specify the commitment of each programme as well as mechanisms to operationalize them.

It will be essential that such partnerships be included both in the regional development programme and in communal development plans, so as to ensure coordination of interventions, optimal use of available financial and human resources in the region and communes, and, ultimately, ownership of achievements by regional and communal authorities. The region and its public technical services, on the one hand, and the communal development committees, on the other hand, must play an active role in the formulation and negotiation of partnerships, and in the coordination and implementation of local development projects and interventions of sectoral programmes at the regional and communal levels, respectively.

Recommendation No. 2 (Government, Regions): Promote the diversification, integration and competitiveness of agricultural activities, by promoting a diversified and integrated value-chain approach, while placing women and youth at the centre of this development effort.

As demonstrated by the project, the value-chain approach has the advantage of addressing both the technical aspects of production and the supply of inputs, processing and marketing of products. However, the development of value chains, already initiated in the project area and focused mainly on three irrigated crops (rice, onion and garlic), should be diversified in such a way as to achieve a better balance between irrigated crops, rainfed crops and livestock, in order to reach the maximum number of farmer categories, reduce risks related to the market and the natural environment, and achieve the various production objectives of farmers (food security, income generation, social status, etc.).

Better integration must be sought both vertically by acting on all the links in the chain, with particular attention to the supply of agricultural inputs and equipment, and horizontally by better integrating value chains with each other. This integration should enable more efficient use of productive assets available at the household level and to take greater advantage of complementarities between value chains, especially with regard to fertility management (crop rotations, crop-livestock integration etc.).

The Regional Rural Development Directorates should continue efforts to identify and promote promising new value chains both for irrigated and rainfed agriculture (potato, tomato, maize, cassava, sorghum, etc.) and for livestock (fattening, animal traction, diary). Regional studies of agroecological, financial, and human and market potential should guide the choice of commodities to promote and ensure better coordination with the opportunities offered by national and regional strategies, especially in the field of agribusiness and export. The CAS has a very important role to play in terms of information transfer, upwards on local conditions and downwards on the evolution of market demand and production techniques. It must facilitate contact between producers and their organizations, on the one hand, and funding sources, input suppliers and agricultural produce markets on the other hand.

Agricultural development should primarily benefit women and youth through well-targeted actions aimed at improving and securing their access to productive resources (land, water, inputs, capital) and at strengthening their technical and organizational capacity.

Recommendation No. 3 (IFAD): To promote flexibility in project design and implementation for the refinement and adaptation of the implementation modalities in response to the changing context, to achieve and maintain a good level of performance (relevance, effectiveness and efficiency) and increase ownership of projects by their stakeholders.

A project's design documents, loan agreement and subsidiary agreements should be sufficiently flexible to enable project teams and implementing agencies to adapt working modalities to the realities and socio-economic, agro-ecological and institutional developments in the project area, while ensuring that that project objectives and target groups agreed between the Government and IFAD are reached. When starting up any project, a summary of project documents in the national language should be delivered to representatives of the target population.

The participatory approach to supporting rural development through micro projects enables target populations to determine the focus of project support for themselves, and to be masters of their own development. However, it is important to provide appropriate advice and support for the preparation and implementation of micro projects. In future, this support should be provided by the CAS. Furthermore, subsidies to farmer groups can be justified as a first step to starting up or intensifying an economic activity, but should be gradually replaced by resources mobilized by the population and through individual and group credit.

The supporting role of supervision missions in project implementation should be strengthened. This would enable them to provide more methodological and technical support to project teams to help them interpret and, if appropriate, adjust project design in order that implementation arrangements and approaches may be developed to optimize the performance of project components. It would also be beneficial to increase the number and weight of project reviews during the implementation period (usually known as 'mid-term reviews'), which constitute the most appropriate occasion to jointly evaluate (Government, project team, IFAD and possibly co-sponsors and the cooperating institution) project performance, identify major constraints to performance and make the proper adjustments to implementation modalities.

A reporting and M&E system is an essential management tool that allows, by analysing successes and failures, for timely and well-informed decision-making in the event adjustments are needed to project approaches or implementation modalities. It is important that, in addition to evaluating only the implementation status of programmed activities, the project's M&E system should also capture the relevance of interventions, level of achievement of objectives (effectiveness), ratio results/costs (efficiency) and impact of the project on the living conditions of the rural population. The system must become functional as early as possible in the life of the project.

Recommendation No. 4 (Government, Regions and IFAD): Consolidate quickly the dynamics of social and economic development already well engaged by PHBM II in its area of intervention, to sustain and enhance the achievements of the project. 

Such consolidation, which is justified by the few shortcomings and risks hovering over sustainability as mentioned above, should be initiated by the region and its public technical services, possibly with the support of IFAD and in partnership with relevant sectoral programmes. It will be necessary to:

• Strengthen the capacity of technical services and the CAS to take over key project functions, ensure the consolidation of project achievements and strengthen the capacity of communes and farmer organizations, including the young umbrella organizations. This will be a priority of the forthcoming AROPA³ project. It will be for the Regional Directorate for Rural Development to foster collaboration between regional partners in terms of rural extension and coaching.
• Complete the rural roads programme, primarily through: (i) upgrading the "black" section of 20 km and the destroyed river crossing on inter-provincial road no. 107 in the south of the project area; and (ii) linking up inter-provincial road no. 117 with national road no. 13 in the west. It will be most important also to establish an effective maintenance system for inter-communal and communal tracks, under the responsibility of the communes, using local microenterprises for routine maintenance and the Road Maintenance Fund (RMF) for periodic maintenance. With support from RMF, the region should assume full responsibility for the maintenance of inter-provincial roads. Roadside dwellers (users' associations of tracks and roads) should be involved only for emergency works (accidental interruptions etc.).
• Promote sustainable agricultural production systems in accordance with national policies and strategies through: (i) better water management at the irrigation scheme and plot level; (ii) promotion of adequate fertilization and crop rotations on irrigation schemes, possibly by introducing horticulture; and (iii) taking more account of the risks associated with recurrent drought, by developing and implementing specific, integrated strategies for adaptation to drought (water and soil conservation, integrated water resources management at the watershed and micro watershed levels, securing fodder, etc.).
• Continue improving basic social services (education, health and drinking water mainly) in partnership with the relevant departments and programmes.
• Consolidate the rural finance system by upgrading the technical and management skills of staff; sensitizing farmers to microfinance through exchange visits, information days, etc.; and promoting savings (especially among herders) by means of specific tools and mechanisms and by mobilizing, through novel microfinance products, the enormous potential for savings and credit offered by farmer organizations.

IV.  SIGNATURES

The Government of Madagascar and the International Fund for Agricultural Development have approved the findings and recommendations presented in this Agreement at Completion Point, based on the Completion Evaluation of the Upper Mandraré Basin Development Project, Phase II. They undertake to implement the recommendations through the IFAD country programme and, in particular, through the forthcoming AROPA project.

This agreement was signed on 5 December 2008 in Antananarivo by Mr. Philibert Rakotoson, Secretary-General of the Ministry of Agriculture, Livestock and Fisheries, and Mr. Benoit Thierry, IFAD Country Programme Manager responsible for Madagascar.

---

1/ Financed by the World Bank.

2/ Evaluation criteria are scored from 1 to 6, a score of 6 being the highest.

3/ Farmer Organisations and Agricultural Services Support Project, cofinanced by IFAD.