Project Performance Assessment

Background. In 2010, the Independent Office of Evaluation of IFAD (IOE) undertook a pilot project performance assessment (PPA) of the IFAD-financed Yarmouk Agricultural Resources Development Project (YARDP) in the Hashemite Kingdom of Jordan. The PPA is a project-level evaluation conducted as a next step after a project completion report validation (PCRV). The PCRV consists of a desk review of the project completion report (PCR) and other available reports and documents. A PPA adds country visits in order to complement PCRV findings and fill in selected knowledge and information gaps. Both the PCRV and the PPA apply the evaluation criteria outlined in the IOE Evaluation Manual but they do so in a selective manner in view of the time and resources available. In particular, the PPA is generally not expected to undertake quantitative surveys. Rather, the PPA adds analysis based on interactions with country stakeholders, direct observations in the field and qualitative information drawn from interviews with beneficiaries and other key informants.

The project. The objective of the YARDP was to improve the food security and income levels of target farmers by arresting degradation and restoring soil fertility for sustainable use of land and water resources through: (i) technical and financial support for the target group to put soil and water conservation (SWC) measures in place and improve agricultural production; (ii) promotion and credit-funding of on- and off-farm enterprises; and (iii) strengthening the capacity of the agricultural directorates in the project area to provide the required technical support services and extension. Total project costs were estimated at US$28.1 million. Of these, IFAD loan was of US$10.1 million. A total of US$12.6 million was to be provided by the Arab Fund for Economic and Social Development (AFESD) but this never materialized. New cofinancing was mobilized from the Abu-Dhabi Fund and the Organization of Petroleum Exporting Countries Fund – US$5 million each. Total project cost at completion was 25.6 million. The initial years of implementation were characterized by a slow rate of loan disbursement caused by technical, policy and financial constraints including those following the removal of AFESD financing. At the original loan closing date (31 December 2006), total disbursement of IFAD loan was at 41 per cent. Thanks to a two-year extension, the IFAD loan was fully disbursed but this required the retroactive increase in the share of civil work expenditures eligible to be financed by IFAD loan from 15 to 40 per cent.

Relevance. The YARDP was aligned with the IFAD 2000 and 2007 country strategic opportunities programmes (COSOPs). Constraints were faced in the application of the proposed integrated ridge-to-valley approach. The Government of Jordan was not fully convinced of the value added of mobilization campaigns through NGOs that were conceived in project design as the instrument for ensuring broad-based participation. The fragmentation of land holdings, the high number of absentee farmers and the tenure arrangements in project area further increased the difficulties to apply the proposed approach. Because of these difficulties, the YARDP followed a pragmatic approach and supported the farmers that complied with the eligibility criteria and that could provide the financial contributions specified in the YARDP appraisal. No information is available on the socio-economic profile of YARDP beneficiaries of SWC initiatives. The only activity with an explicit social focus was the programme income-generating activities (IGA) loans targeted to women. All in all, the YARDP did not show a clear conceptual linkage between the characteristics of the disadvantaged rural population, as described in design, and project activities.

Effectiveness. The subsidized financing of on-farm SWC was the most important instrument mobilized by the YARDP for achieving its objective. In this sector, the most serious constraint was convincing farmers to invest in SWC in agriculture and dedicate more time and effort in agriculture activities. The project proved that with sufficient awareness and financial incentives, farmers were willing to invest their own resources in land development. Despite the good result achieved, the size of land under improved SWC practices was 56 per cent of the target (84 per cent compared to revised targets). The PPA confirmed the effectiveness of IGA loans for initiating or expanding micro enterprises often led by women. On the contrary, project investments in SWC did not result in an increased demand for credit as envisaged by YARDP designers. Despite the training programme sponsored by the YARDP, the quality of agricultural extension remained challenging.

Efficiency. The withdrawal of AFESD stalled project investments in early project years. As a result of the slow disbursement, the extension of the loan for two years inevitably increased expenditure on management and supervision. The YARDP efficiency performance benefited from the cost-saving capacity of project implementers. Total management costs were estimated at 16 per cent of the total cost, which is in line with the average of IFAD-financed operations. In the PCR, the economic internal rate of return (EIRR) is estimated at 20 per cent that favourably compares with the 17 per cent at appraisal. Although, this estimate could not be verified, the PPA confirmed that the resources invested under the project in orchards and SWC measures are likely to generate significant and enduring benefits to beneficiary farmers that will more than compensate the resources being invested.

Impact. The YARDP rural road rehabilitation programme (287 kilometres) positively affected the living standards of 13,000 households. Road construction caused an average financial saving in transportation for beneficiary households totalling JOD 105 per month and increased by 50 per cent the value of houses. Farmers benefiting from SWC measures reported an income increase of 20 per cent. The income generated by investments in IGAs was estimated between US$2,500 and US$4,200 per year. According to the project impact surveys, the average monthly income of YARDP beneficiaries increased from JOD 298 to JOD 392. The average contribution of farm income to total household income increased from 25 to 57 per cent.

Annual agricultural income increased from US$1,260 to US$3,780. It is however important to consider that the methods adopted in the project impact surveys did not explicitly capture the extent to which these results are attributable to the YARDP. The project made an effort for capacity building of rural women although this was not sufficient to address the lack of technical and business capabilities affecting smallholders in the project area. With regard to social capital, the project was expected to generate a significant impact on the development of groups. This objective clashed with the individualistic predisposition of highland farmers and the difficulties faced by the YARDP to undertake comprehensive participatory campaigns. In terms of agriculture development, the project provided incentives to farmers to shift from cereal crops to more productive land-use systems based on perennial fruit trees and olive trees. The on-farm water harvesting structures (cisterns and stone tree basins) had a positive impact on land productivity that increased by approximately 20 per cent.

The income increase had a positive effect on the capacity of beneficiary households to purchase food. The expansion of livestock production and dairy processing activities financed through IGA loans implied higher availability of food items within the households.

Nevertheless, evidence is available of an increased number of households experiencing hungry seasons and a 4 per cent increase in acute malnutrition. These differences may be explained with the increase in price of food staples and other products. Unfortunately, no information is available on the extent to which these changes affected YARDP beneficiaries. At the same time, the characteristics of the survey methodology do not allow to draw any conclusion on the casual linkage between the YARDP and the above hunger and malnutrition facts. In terms of natural resources, productivity of land increased by 29 per cent in sites that benefited from on-farm SWC.

The greatest merit of the YARDP was indeed to increase the interest of farmers in land conservation that may generate important results in terms of environmental conservation and development. The patchiness of SWC measures and the fact that the integrated ridge-to-valley approach was not implemented reduced the net impact of the YARDP that is however positive.

The most important institutional change of the project was the establishment of the SWC units in the Agricultural Directorates of Irbid, Bani Kenanah and Ramtha. Despite the fact that the YARDP made an important effort for capacity building, the funds available were not adequate to generate visible changes in extension service.

The YARDP had also limited leverages on the technical capabilities and/or the institutional functioning of the Agricultural Credit Corporation (ACC) and the training programme for the ACC cadre was not implemented. The most important policy change was the decision from the Government of Jordan to continue the stream of activities initiated by the YARDP for two additional years.

Sustainability. The sustainability of micro enterprises supported with IGA loans is significantly exposed to market-related risks as confirmed by a publication by KariaNet ¹ where it is noted that 39 per cent of the women that took a loan under the YARDP could not continue the implementation of the project because of marketing obstacles and the inability to market their products. Market-related risks include the highly volatile price of inputs, the difficulties in accessing storing facilities or alternative sale channels, the absence of instruments for coordination with buyers and for information sharing, the weakness of existing cooperatives. The sustainability of the off-farm SWC structures is meagre and the project did not devote significant efforts to the maintenance systems for off-farm SWC structures.

Innovation. The YARDP design included several innovative features but – as noted in the PCR – the project was not effective in mainstreaming such innovations. The communities targeted by the YARDP did not embrace the idea of a collective management of off-farm natural resources. The project however succeeded in promoting a new model for promoting SWC investments compared to past experiences based on a charitable approach.

With regard to the establishment of users' groups, its implementation became difficult because of the low emphasis on participatory approaches and the overall slow rate of implementation of off-farm investments. At the end, very few water groups were formed. The third innovation (establishment of a credit line with a focus on women for IGAs) was implemented but not in line with the modality described at design: instead of the NGO, it was the ACC that was responsible for the management of this credit line.

Scaling up. The PCR provides very limited information on the scaling up performance of the YARDP. Building on YARDP experience, the Ministry of Agriculture (MOA) has established a land development programme and a water harvesting programme; a programme for income generation (the Hakoora programme) was designed based on the demand for small-scale investments by women and poor rural households promoted by the YARDP. These were however not documented in the PCR or in any other document. IOE could not find evidence of a systematic approach for dissemination of YARDP experience aimed at scaling up. The intended strategy of the YARDP based on an integrated ridge-to-valley watershed management approach can be implemented at national scale if appropriate supporting conditions are in place. If so, the lessons learned of the YARDP and the difficulties faced in applying this approach could be very useful. The prospects of scaling up are inevitably affected by the fact that very few donors operate in the agricultural sector in Jordan.

Gender equality and women empowerment. The IGA loans were the activity of the YARDP with an explicit social connotation. In this, the project promoted a broad participation of women and achievements were above expectations. Women's lack of title to the land and of solid collateral had limited their chances of meeting eligibility criteria for credit. For this reason, the ACC required the formal involvement of a regular income earner in the household as guarantor. It can be argued that such model may not represent a genuine instrument of women empowerment as it required support and acceptance from the regular income earner of the family (that in most of the cases was the husband). Nevertheless, the evidence gathered during the PPA showed how IGA loans contributed to the economic and social empowerment of women. Many of the women borrowers interviewed during the PPA declared that the income gained in the activities financed by the loan became higher than the husband's salary. As a result, women increased their command over household financial resources and benefited from an increased influence in decision-making processes.

Performance of partners – IFAD. The lessons learned from previous development interventions were not effectively addressed in the YARDP design. The project continued to face problems in promoting beneficiaries' participation and the solution proposed (consisting in the recruitment of a specialized NGO) was not accepted by implementing partners. This undermined the application of the key innovative feature of the project. Overall, the quality of project design was affected by the fact that important elements of Yarmouk socio-economic structure (i.e. land fragmentation, individualistic predisposition of farmers, limited share of communal land) were not clearly incorporated in project implementation strategy. The very important role of IFAD during early implementation years should be recognized: the Fund actively and proactively engaged in supervision and performance monitoring. No mid-term review was undertaken but a project review was mounted in June 2006, around the original date of project completion.

Performance of partners - Government of Jordan. This PPA confirmed the positive appreciation of the PCR for the Government of Jordan's strong ownership of YARDP goal and objective. Throughout the implementation history of the YARDP, the Government of Jordan ensured the availability and the timely provision of counterpart funds. AFESD was removed from YARDP financing in line with an executive Government of Jordan's decision on the distribution of international financing among various projects. This decision stalled investments in resources development until the Government of Jordan obtained alternative financing. The programme management unit (PMU) established successful cooperation with the different governmental institutions partners involved in YARDP implementation. The PMU provided effective leadership in coordinating the wide range of activities included in the YARDP. The monitoring and evaluation (M&E) system put in place under the YARDP performed satisfactorily by ensuring the rigorous tracking of project outputs. The system could have been integrated with monitoring of expected outcomes at watershed level or with a continuous monitoring of the socio-economic status of beneficiaries. The ACC has been a very important partner for the YARDP as the activities implemented by the ACC absorbed more than 37 per cent of IFAD loan.

Performance of partners - cooperating institution. AFESD was removed from project financing but remained responsible for supervision of project fiduciary aspects. The appointment of the United Nations Office for Project Services (UNOPS) as cooperating institution in May 2006 resulted in a supervision process that devoted higher attention to the achievement of development objectives. IOE noted significant disconnects between UNOPS supervision ratings and those formulated by IFAD in the annual portfolio review processes. Such mismatch could be due to various factors but it may indicate the fact that the cooperating institution was not very effective in sharing with IFAD its concerns on project performance.

Conclusions. The implementation of the YARDP has been characterized by a very slow disbursement rate and it was only after enforcement of the recommendations of the IFAD project review that the full amount of resources available under the IFAD loans could be disbursed. The decision not to use the service of a specialized NGO for beneficiaries' mobilization combined with structural problems such as the high fragmentation of farm landholdings, the high number of absentee farmers and the complex tenure arrangements further impeded the application of the proposed ridge-to-valley approach. The YARDP implementers identified a more suitable approach based on the voluntary participation of individual farmers. The YARDP succeeded to promote farmers' own investments in agricultural land development and visible results were achieved in terms of expansion of olive orchards and improvement of land productivity. The performance of the YARDP in terms of off-farm SWC structure was significantly below expectations. The disbursement of agricultural loans was below targets. On the contrary, the IGA loans were very popular and positively impacted on household income, livelihoods diversification and expansion of agricultural businesses. These constituted an opportunity for enhancing the role of women in income generating activities and decision making in the household. The financial sustainability of micro and small enterprises is however at risk due to the high exposure to market-related risks.

Recommendation 1

Strategy for agricultural market development. The Government of Jordan should consider (whether in cooperation with IFAD, with other donors or through regular budget) the opportunity for a coherent programme of agricultural market development aimed at promoting the participation of smallholders as well as small processors in urban food markets. This would necessarily require the upgrading of available skills at farmers' level as well as the strengthening of tools for coordination and information sharing among all market actors.

Recommendation 2

Responsibility for monitoring of off-farm infrastructure. The Government of Jordan should determine clear responsibilities for maintenance of off-farm infrastructure. The Ministry of Water and Resources and Irrigation was involved in the design and construction supervision of the recharge dams and other off-farm SWC measures. However, the arrangements for supervision are currently very unclear. The MOA claimed that they would take care of any damages to these structures, but there was no clear-cut process of monitoring or of preventive maintenance. A clear attribution of monitoring responsibility to the Ministry of Water and Resources and Irrigation is therefore needed including with regard to the timing of site inspection, budget and human resources implications.

Recommendation 3

Review of the Results and Impact Management System (RIMS) pilot. The PPA recommends the undertaking of a review of the RIMS pilot surveys. Important knowledge gains could be generated by introducing a differentiation between the treatment and the control group. In projects where the RIMS is accompanied by tailored impact assessment surveys, IFAD and the concerned government should explore opportunities for cost-savings.

---

1 Knowledge Access in Rural Inter-connected Areas Network (KariaNet) is a pilot project financed by the International Fund for Agricultural Development (IFAD), the International Development Research Centre (IDRC) and IFAD-funded projects in the Middle East and North Africa. It aims to enhance networking among IFAD-funded rural and agricultural development projects in the region in order to improve knowledge sharing and information/experience exchange. See Karianet website.