Completion evaluation

Introduction

Evaluation process and objectives. In 2008, the Executive Board of the International Fund for Agricultural Development (IFAD) requested the Independent Office of Evaluation of IFAD (IOE) to undertake the final evaluation of Phase II of the South Western Region Small Farmers Project (PROPESUR) in the Dominican Republic. This evaluation, which was conducted between July and December 2009, focuses on: (i) assessing the performance and impact of PROPESUR, and (ii) identifying lessons to be learned and recommendations that can be used to improve the design and implementation of similar development projects in the country. The evaluation report also seeks to provide inputs for the new IFAD countrystrategic opportunitiesprogramme (COSOP) for the Dominican Republic.

Methodology. The evaluation methodology entailed both a review of the existing literature and work on the ground in the Dominican Republic. Two missions (a preparatory mission and the main mission) were carried out. The field work included a representative survey of 33 community organizations that received assistance under this project and the triangulation of the findings derived from the full range of evaluation data. The final report was then drafted and reviewed by the Latin American and the Caribbean Division of IFAD and by governmental and other project officials. The agreement at completion point is included in this report.

The socio-economic context and rural poverty. The area near the border between the Dominican Republic and Haiti, which is where PROPESUR was sited, is one of the poorest regions in the country. Ever since the 1980s, IFAD has been providing support for projects undertaken by the Government of the Dominican Republic in an effort to promote the development of this zone, in which a high migration rate is combined with the relative absence of a State presence, public services and substandard living conditions.

The project. PROPESUR was designed in 1998 and executed between 2000 and 2007. It was extended on three occasions. The target population was made up of some 10,000 rural households. PROPESUR's objectives were to boost income levels, improve the living conditions of poor farmers and alleviate that population group's poverty by promoting the sustainable social and productive development in the project area. Project design included two programmatic components: community development and access to rural financial services. PROPESUR proposed a number of innovations, such as demand-driven planning/operations, the close involvement of the private sector in its execution, measures for strengthening rural financial services and emphasis on gender equity. The total project budget amounted to US$17.6 million and its budget execution to US$15.84 million.

Project performance

Relevance. PROPESUR was relevant within the context of the prevailing situation in the country. The changes made in the design of PROPESUR based on the outcomes of the previous project proved to be more effective in reducing poverty, despite the risks that were taken on board. In the course of the project's implementation, IFAD's approach evolved and matured as the result of a conceptual shift that expanded upon the focus on poverty reduction to include a human-rights- and citizenship-based perspective. This paved the way for a fuller understanding of how much social – in addition to economic – factors influence living conditions and for the introduction of the dimension of food security. New methods for gauging poverty levels were also introduced through the Results and Impact Management System (RIMS), and these changes influenced the direction taken by the project.

As far as the project design is concerned, the methodologies used for various courses of action and subcomponents – based on different approaches to the wide range of activities involved in servicing different categories of beneficiaries (communities, organizations, households, individuals) – were quite sound. The challenge that this posed for the project management team resulted in some difficulties in starting up the project on the ground, however. In its initial phase, emphasis was placed on social projects, to the detriment of strategies for providing appropriate support along the entire length of the production chains targeted by the project (coffee, bananas, sheep and goats).

Effectiveness. Grass-roots organizations were able to build their capacity for self-organization and project management under adverse conditions (absence of a State presence, emigration of young people, illiteracy, etc.), and gender mainstreaming in the various project areas was successful. Outsourcing to private groups proved to be an effective promotional means of including community organizations as project participants and executing agencies. By the same token, intermediary financial institutions (IFIs) that received support were strengthened, and the project succeeded in introducing additional professional capacity in their zone of influence. What is more, the project's social investments significantly improved the quality of life in the target communities. On the other hand, support for agricultural production was begun too late to permit the sustainable integration of a majority of the target households into the production chain, and the project was therefore not very effective in this respect. It made credit more accessible to a substantial number of poor clients on better terms, but the tie-in of this component with community development efforts remained weak. The project did not succeed in promoting the start-up of new rural microenterprises and was not effective in promoting sustainable natural resource management.

Efficiency. Between the project design stage and its launch, a number of changes occurred in the situation with respect to the dismantling of the decentralization strategy that made the project's implementation less efficient. It took a great deal of time to adapt the project to these changed conditions and for it to "take hold" on the ground. Its operating costs were high, with a third of the resources being used for the management and administration of a project that was outsourcing services via external coexecuting agencies. In addition, the expenditure on technical assistance for the development of production activities was excessive in view of the results obtained. There was an evident imbalance between sub-execution arrangements for programmatic areas such as community investment and credit funds, on the one hand, and expenditure on management, administration and outsourced operators and services, on the other. The cost of the area promotion agencies (APAs) relative to the institution-building effort was reasonable.

Impact on rural poverty

In the area of income levels and net assets, the project raised the incomes of 430 coffee-producing families, and the prospects for sustaining that increase are good. Most of the over 4,000 IFI clients expanded their operations and sales. The latrines that were built resulted in an increase in the assets of over 2,400 households, while the other community infrastructure works remained in the hands of the State. The project made it possible for the NGOs that managed the APAs and IFIs to increase their assets and incomes on a sustainable basis.

The project succeeded in considerably strengthening the administrative and management capacity of the grass-roots organizations that it supported, but without increasing the cohesiveness of the communities concerned to any substantial extent. Gender mainstreaming efforts had a notable qualitative impact in terms of the women's self-esteem and of how the roles of men and women are viewed, including within the family. The boards of directors of mixed organizations have not yet given women a greater role, however. Access to financing had the effect of obtaining greater recognition for female clients, both within the family and in their neighbourhoods.

With regard to agricultural productivity, considerable gains were seen in quality coffee and organic banana crops which can potentially have a significant impact. No sustainable improvements were achieved, on the other hand, in livestock activities (sheep and goats) or in terms of the provision of training in dry forest management techniques. The steps taken to increase food security did not result in a noticeable change in the quality or quantity of food consumed by families in the target group.

Project efforts to improve natural resource management had an impact at the household level (latrines, use of agrochemicals), but not in terms of the integrated management of microcatchment areas. Finally, at the institutional and policy level, the fact that the project worked fairly independently had the effect of strengthening the private institutions involved, whereas the integration of stakeholders with public agencies was hampered by the lack of appropriate policies and structures, although those components are now starting to be developed.

Other performance criteria

PROPESUR furthered the application of more effective forms of action and of a viable model for channelling Government funds to a marginalized, rural target population. Reinforcing grass-roots organizations and communities and working directly with them heightened the social sustainability of project interventions. In the economic and financial sphere, facilitating commercial-crop producers' integration into production chains helped them to become self-reliant, as did the work with profitable local IFIs.  The outlook for the sustainability of project interventions focused on food security and the environment is poor, however.

The project introduced a number of major innovations: coexecution in partnership with private NGOs in building organizational capacity at the community level; the combination of a demand-driven approach to planning/operations for grass-roots organizations or communities with the analysis of organizational, local and regional potential; greater gender equity in all spheres; and the provision of access to rural financing using an approach that differentiated among various areas of activity, rationales and instruments. While these experimental initiatives are replicable and can be expanded upon, they have not been taken on board or disseminated within State institutions, private-sector enterprises or international cooperation agencies. Thus, these innovations have, as yet, neither been replicated nor expanded upon in any significant way.

Performance of partners

IFAD's performance in the implementation of PROPESUR merits recognition in view of the fact, first of all, that it piloted the direct supervision approach ¹,which is a model in whose use IFAD has had only limited experience. It continued to use monitoring tools such as on-site biannual visits and project status reports and was able to follow the project closely through the Senior Technical Advisor, who was well versed in IFAD standards. The Government counterpart, Oficina Nacional de Planificación (ONAPLAN²) values the role played by IFAD in providing constructive support and cooperation. IFAD also made good use of the technical resources available from regional programmes such as, for example, FIDAMERICA (Network of IFAD-supported projects in Latin America and the Caribbean) and PREVAL (Regional Platform for Evaluation Capacity Building in Latin America and the Caribbean). Nevertheless, from a strategic standpoint, the guidance provided by IFAD in the course of the project was rather limited, particularly in view of the proposed innovations, the project's relative institutional and geographic isolation, and the changes that were made, both in the project environment and in IFAD's project concept. The difficulties that the project had in "taking hold", frequent changes in the logical framework, and the limited implementation of project management and administrative measures called for decisions at the supervisory level that were not made. The mid-term review did recommend major adaptations, but it should have been done a year earlier in order to rectify the delays that occurred in implementing the productive development support strategy; this would have saved time, which would have been of crucial importance in achieving better results. At project completion, suitable exit strategies were lacking in the production and microfinance components of the project, and the valuable experiences and lessons that could have been learned from PROPESUR have not been systematized or disseminated.

The Government of the Dominican Republic followed the PROPESUR pilot with interest and facilitated its operation with capital inputs. The executing agency (ONAPLAN)

played an active part in project management, although its capacity to provide guidance and strategic oversight was limited by the fact that it does not have a great deal of expertise in rural development. While the continuity of professional project staff was maintained, various changes were made in the management team (a critical factor, given the innovations proposed by the project and the strategic weaknesses mentioned earlier) during the first phase of the project which undermined its leadership capacity. In the second project cycle (2005-2007), the management team improved substantially, thanks to the arrival of a new manager with the necessary experience and qualifications. The Government honoured its financial commitments despite the difficulties it had in doing so owing to the economic and financial crisis that broke out in 2002. At times the funding for project activities was late in arriving, particularly at the start of each year and after changes of government administration.

The coexecutors – associations formed by members of the target group, NGOs, IFIs and other service providers – exhibited a strong commitment to the project and performed well. The performance of the APAs was particularly noteworthy.

Overview of Ratings

The following table provides an overview of the project's ratings based on the various evaluation criteria:

 

Evaluation Criteria	Ratings*
Core performance criteria
Relevance	5
Effectiveness	4
Efficiency	3
Project performancea	4
Rural poverty impact	4
Household incomes and assets	4
Human and social capital and empowerment	4
Food security and agricultural productivity	3
Natural resources and environment	3
Institutions and policies	4
Other performance criteria
Sustainability	4
Innovation, replication and scaling up	4
Overall project achievementb	4
Performance of partners
IFAD	4
Government	4
Coexecuting agencies	5

a  Average of ratings for relevance, effectiveness and efficiency.
b The overall project achievement rating is calculated based on the ratings for project performance; rural poverty impact; sustainability, innovation and scaling up.
* Ratings are assigned on a scale of 1 to 6 (6 = highly satisfactory; 5 = satisfactory; 4 = moderately satisfactory; 3 = moderately unsatisfactory; 2 = unsatisfactory;1 = highly unsatisfactory).

Conclusions

PROPESUR combined the promotion of rural production activities with an approach designed to involve beneficiaries in setting priorities, project execution and monitoring. The innovative project design left some aspects to be defined or fine-tuned in the course of implementation within a changing project environment. As a result, PROPESUR needed a fairly long time to consolidate its strategy on the ground, especially in the case of its production component. In this kind of trial-and-error situation, it is crucial to have oversight at a higher level at which decisions can be taken as to the project's strategic direction. There was not enough of this sort of guidance in PROPESUR's case.

Although the project did target the poor population, this group is composed of a number of very different segments. Effective management requires that specific strata be selected as the target group for project efforts.

The project was effective in various areas (grass-roots organization and gender equity, service delivery, social infrastructure, access to financial services) and less so in others (production and environment, food security). The synergies between (sub)components were more modest than expected. The work done in order to strengthen commercial production chains needs to be completed in order to integrate the producer associations into the more profitable chains. It is of critical importance to learn the lessons to be drawn from this experience in order to combat poverty in this zone more effectively.

A comparison of the total investment in this project (US$16 million) with its results reflects the cost of the delay experienced in getting it onto a solid footing. The efficiency of resource use was diminished by a lack of clarity at the strategic level.

For a number of sound reasons, as well as because the State maintains a very limited presence in the area, this project was given a great deal of scope for experimentation. Since support for rural development in this zone is to be continued, further progress needs to be made in increasing inter-agency cooperation, both within the public sector and between the public sector and the private sector.

Although some efforts have been made in this respect, room for improvement exists with regard to the dissemination of these experiences and the lessons learned about development in the zone, particularly in view of the highly innovative nature of PROPESUR.

Recommendations

PROPESUR closed its doors in mid-2008. In the interim, the Government of the Dominican Republic and IFAD have agreed to prepare two new rural development projects, one of which is to be sited in the same zone along the border with Haiti (although it will cover a larger area than PROPESUR did), and the new COSOP was approved in April 2010. The evaluation team is making the following recommendations for the future projects in the country to be pursued under the new COSOP.

(R1) Ensure that the institutional/political anchor for the new projects is an institution that specializes in rural development within the framework of the Government's new policies and strategies in this area of endeavour. Interaction with various public bodies at the central and municipal levels, with other development projects and with private organizations should be scaled up with a view to identifying possible synergies and establishing a more solid anchor for activities in the new project areas.

(R2) Design and targeting:

Improve the definition of the target population by indicating the types of units that the project will focus on (households, individual microenterprises, grass-roots organizations, communities), and ensure that there is a clear differentiation of the "instruments-interventions-time horizons" cluster based on the differing needs and capacities of the various target groups and their differing approaches to the dynamics of progress;

Ensure that project beneficiaries actively participate in the definition and implementation of lines of action throughout the project. This should be a demand-driven process, but other elements should also be taken into consideration, including the potentials and risks associated with the various organizations and with the region, the environment and the market;

Bear in mind the provisional nature of some aspects of the initial project guidelines. A distinction should be made between the mandatory components of the project (commitments made in the loan agreement) and those that are indicative and subject to the determination of those responsible for its implementation (steering committee and management). It is recommended that implementation guidelines to be followed by executing agencies be pre-defined in the loan agreement³

(R3) Implementation arrangements:

Ensure that tasks of strategic importance for project implementation are performed by organizations with the necessary experience and capacities. These tasks include: supervision of operational and financial execution, monitoring of implementation and decision-making with regard to the project's direction (especially when significant changes take place);

Examine the approximate costs of different implementation models (execution by project, outsourcing of services, or a combination of the two) in order to establish suitable guidelines;

Plan for the appropriate outsourcing of technical/financial services. The utilization of existing capacity in the area will require the use of an explicit negotiating strategy, in particular for dealings with the major NGOs that are active in each territory.

(R.4) Lines of action/components:

• Complete the activities undertaken to support commercial crops (e.g. coffee and bananas) grown by farmers' groups at all stages along their production chains ⁴ and revisit the range of potentially job-creating rural production activities to be promoted, including irrigated farming (as outlined in the design of PROPESUR);
• Promote the use of competitive project funding applications as a means of financing productive projects at the community leveland encourage young people to take part in these processes. It is recommended that an effort be made to encourage innovative ideas and the development of economic relations between communities and urban areas;
• Evaluate the stage of consolidation reached by the social organizations for which support is to be provided as a basis for determining the specific kinds of assistance to be supplied by the project in each case ⁵;
• If the decision is taken to continue to support the construction of social infrastructure works, explicitly define the relationship between community representatives and the grass-roots organizations involved with regard to the division of responsibilities for maintenance and possible future extensions or expansions;
• The nexus between the provision of access to financing and the demand for credit in rural areas should be (re)considered, and support should be provided for microfinance initiatives that are more solidly rooted in the rural communities concerned. Weather insurance instruments merit support, and thought should be given to ways of contributing to the development of a model that could be applied nationwide (rather than a service tailored exclusively to project clients while they remain in the project area ⁷);
• In the area of food security, it is recommended that investment be directed towards the design of a programme focusing on improvements in the medium and long terms, rather than the short run, ⁵ and that the possibility of combining this line of action with Government subsidy programmes be explored⁸.
• With regard to knowledge exchange, the evaluation team recommends that the systematization of the PROPESUR experience be completed and that this information be disseminated and exchanged with specialized stakeholders (NGOs, public stakeholders, projects) in order to support good practices and contribute to lessons learned.

---