Completion evaluation

The core learning partnership and the users of the evaluation

In 2006/2007, IFAD's Office of Evaluation (OE) conducted a completion evaluation of the Apuseni Development Project (ADP) in five judets (administrative districts) in Romania. An evaluation approach paper was prepared in June 2006 and discussed with the Fund's Near East and North Africa Division (PN) and with the Government of Romania (GOR). As per usual practice, an evaluation core learning partnership (CLP) was formed comprising representatives of the Ministry of Public Finance (MOPF), the Ministry of Agriculture (MOA), the Cooperating Institution [United Nations Office for Project Services (UNOPS)], PN and OE.

The main evaluation mission took place between September and October 2006, in concomitance with a small-scale survey in the field to complement the existing data set. The survey was commissioned by OE and conducted by an independent consultant based in Romania. A draft evaluation report was distributed in September 2007, and a final workshop was organised in Bucharest on 3 December 2007 to discuss the findings from the evaluation and lay the basis for its Agreement at Completion Point (ACP). The meeting was attended by members of the core learning partnership and other key stakeholders. The ACP illustrates the Government of Romania and IFAD's agreement on the main findings from the evaluation.

Main evaluation findings

Implementation progress

For the first half of its life, ADP was essentially a non-performing agricultural credit project, despite vigorous efforts to stimulate it into life. The project became effective in November 1999, but by the time of the first project Supervision Mission one year later, no loans had been disbursed. The second Supervision Mission stated that the project remained "in a state of crisis" as only three loans had been approved 20 months after loan effectiveness and twelve months after the signing of the Subsidiary Financing Agreement with the sole participating credit institution, a state-owned commercial bank. Three years after the project launch, there was a marked improvement in loan uptake, though at the two thirds point through the project, only 16 per cent of the IFAD loan had been disbursed. Disbursement activity was sustained over time through a combination of loan extension, expansion of geographic coverage, and through a fundamental shift in project strategy towards a growth-led paradigm and larger loans.

Project relevance and design 

The project's development objective was highly relevant to the needs of rural communities in mountainous areas. At the time of design, two fifths of the rural population was living below the poverty line of US$3.30 per day, and the mountainous areas were judged to be particularly disadvantaged. Yet, the project presented faults at the level of immediate objectives and operational design. The rationale for involving the poor in a formal credit arrangement was not spelt out and the target groups' "effective" demand was uncertain. The modalities and conduits for dispensing credit to small borrowers were quickly proved not to be viable. Over time, the project design has become more relevant to an environment which is macro-economically and institutionally more stable, and government rural development policies which are aligning to European Union (EU) membership, and large-scale enterprise development. The introduction of major design changes in the project in 2003 addressed in part the original design issues and reinstated the relevance of the project.

Impact

Overall, there is strong evidence of improved productivity for the non-poor individual and corporate borrowers who now constitute the primary beneficiaries of the IFAD loans. Many of these are early movers taking advantage of the post- communist era vacuum of appropriately-scaled rural technology for a market economy.  The spread of benefits is narrowly confined. ADP has impacted on about 1,000 non-poor borrowers, the primary beneficiaries who will have most to gain in term of the ability to build household physical, financial and social assets. However, the investment activities of the primary borrowers has created around 1,300 jobs which are mostly taken up by the unemployed, who are likely to be asset and income poor. In addition to this group of poor secondary beneficiaries, there is a group of tertiary beneficiaries in the dairy sector. These processors have provided an assured market for about 700 non-borrower milk producers for whom an estimated 1,050 further jobs have been ensured or created. About half of these jobs are in households owning less than five cattle, who are categorized as poor.

Thus, about half of these jobs would be for family labor in poor families. Thus in total, a range of 1,900 to 2,300 jobs have been created for the poor, compared with an original target for poor beneficiaries of 6,000 – 6,500.

Sustainability and ownership

The rating for this is determined by whether the results of the project will be sustained in the future without external assistance. Bearing in mind that the results of the project have limited impact on the poor, the assessment is nevertheless that some supporting factors (ownership, technical and financial sustainability) make the prospects for the systems put in place by the project in the medium term quite good, even though the future of the Revolving Credit Fund remains uncertain.

Overall assessment

ADP was designed and implemented under difficult conditions of macroeconomic and financial sector volatility, and institutional uncertainty. However, the evaluation raises a number of questions about the design system that is operated by IFAD. These relate to the trade-off between maintaining continuity between the different phases of design, and the objective evaluation that is needed at appraisal to confirm the main design parameters and assumptions of the project.

This evaluation points to the major shift in the project lending strategy from a small- to a medium- and large-size borrower emphasis in line with the evolution of IFAD's understanding of rural finance needs in transitional economies. It also points out that the impact paradigm that was implicit in moving from smaller to larger borrowers, and the assumed benefits from backward and forward linkages, were not adequately analyzed as a basis for the shift in strategy. Nor was an associated growth-based operational strategy put in place to ensure that poverty impacts would be maximized. And given the fundamental change in the impact paradigm of the project, there should have been a formal re-appraisal.

Recommendations agreed upon by all partners

There are no specific recommendations for the project per se since further IFAD loans are not in the pipeline, given the fact that Romania has become a part of the European Union. However, the following recommendations are offered to IFAD and the Government of Romania for the design and implementation of future projects and programmes similar to the ADP.

Design Process

The evaluation has highlighted a number of issue areas in the IFAD project design process. These relate to the appraisal function, to the internal accountability process involving the Technical Review Committee (TRC) and the Operational Strategy and Policy Guidance Committee (OSC), to organizational incentives relating to disbursement as the driver of performance, and to the issue of re-appraisal of a non-performing project.

IFAD is currently revisiting its overall design process through the Action Plan to ensure the quality and enhancement of its products, performance and processes.

Within this context, the Fund should:

• Review the role of appraisal in the overall design process, distinguishing between its use for "critical assessment" or for "confirmation" of design parameters and assumptions
• Ensure an appraisal function which is independent of formulation
• Review the manner in which comments provided by TRC and OSC are dealt with in relation to project appraisal and finalization, possibly having OSC comment at the appraisal stage
• Review organizational incentives relating to disbursement as an implicit driver and measure of portfolio and loan performance; and
• Consider a trigger mechanism for automatic re-appraisal of non-performing projects, possibly based on supervision performance ratings

It is expected that some of the above recommendations will be addressed by the new independent Quality Enhancement mechanism (already operational) and the Quality Assurance mechanism (recently established in January 2008) which will come under the responsibility of the IFAD Vice-President's Office.

Partners involved: IFAD

Targeting and monitoring and evaluation

Several findings on the issue of Targeting and Monitoring and Evaluation (M&E) stem from the evaluation of the ADP. Recommendations to this regard include the following:

• The need for IFAD to develop a clear target group at the stage of project/programme design in a defined area and remain focused in line with the provisions in its targeting policy
• Consider, where appropriate, the undertaking of a baseline before implementation start-up
• Identify quantitative and qualitative Objectively Verifiable Indicators (OVIs) and include these in the log-frame; and
• Ensure that an M&E system is a core activity of any project

Partners involved: GOR, IFAD

The Revolving Credit Fund

One question raised by the evaluation with regard to the sustainability of the credit component relates to the future of the Revolving Credit Fund, that is, to ensure that it will continue to revolve as planned until 2018.  The project partners seem willing to continue using the revolving funds. This implies that lending will be on a declining scale, given the schedule of repayments to IFAD and the fact that there has been no contribution of own bank resources. Representatives of Banca Comerciala Romana (BCR) confirm that they will stay with the project until its end and will utilize all the remaining available funds. Furthermore, BCR expects to expand its rural portfolio, building on the experience with the IFAD loans when using EU restructuring funds in all its branches; Banca Comerciala Carpatica (BCC) has also been very active in this regard and has utilized access to ADP funding to support the expansion of its branch network.

Based on the above, the following recommendations pertain to the Revolving Credit Fund (RCF:

• Avoid extending the RCF to additional banks; and
• The PCIs should consider leveraging any funds (or funds from other donors) to maintain RCFs and/or develop new product or services of relevance to the target group

Partners involved: GOR, PCIs

Recommendations not agreed upon by participants

The following recommendations, as contained in the completion evaluation report, were included in the draft issues paper submitted to workshop participants for consideration. Following deliberations at the meeting, it was decided not to have them included in the final Agreement at Completion document:

• Encourage the Participating Credit Institutions (PCIs) to integrate the lending into the bank's regular operations as a loan product or service of the bank itself, without explicit reference to IFAD as a donor.
• The MoPF and participating banks indicated that, in their opinion, the ADP-financed loans were integrated in regular operations, as the decision making mechanisms followed regular bank procedures. Any integration beyond that implemented would not be possible given MoPF emphasis on the fact that ADP was financed through external lending with a sovereign guarantee.
• Reconsider the requirement to submit every loan application for confirmation of eligibility to MoPF, along with the requirement for every repayment be directly transferred to MOPF; instead, repayments to MoPF should be made in tranches adjusted to the requirements of repayments to IFAD.
• MoPF reiterated the fact that confirmation of eligibility on a loan by loan basis was a sine qua non condition for drawdown of loan funds, given MoPF's final responsibility for the use of IFAD loan funds as per the signed loan agreement.