Interim evaluation¹

Extract of Agreement at Completion Point

The Core Learning Partnership and the users of the evaluation

This Agreement at Completion Point (ACP) is concerned with identifying lessons from the implementation experience of the first phase of the Agricultural Resources Management Project (ARMP) which are applicable to future interventions and registering stakeholders commitment to their implementation. This will be facilitated through the Core Learning Partnership (CLP) which serves as a platform for effective transfer of knowledge and learning generated through evaluation. IFAD's Evaluation Policy² defines the main objectives of the ACP as… "to clarify and deepen the understanding of evaluation recommendations, document those that are found acceptable and feasible and those that are not, and eventually generate a response by the stakeholders on how they intend to act upon them within the framework of an action plan that assigns responsibilities and deadlines." The role of the Office of Evaluation (OE) is to facilitate this process leading to the conclusion of such an agreement.

As agreed during the evaluation process, members of the CLP include representatives from the Ministry of Agriculture, Ministry of Planning, Agricultural Credit Corporation (ACC), the Arab Fund for Economic and Social Development (AFESD) and project and IFAD staff.

The main evaluation findings³

Performance of the Project

Relevance. The evaluation concluded that ARMP's soil and water conservation (SWC) and agricultural development objectives remained very relevant to the intended target groups. This conclusion was derived from considerations of the impacts on poverty presently being achieved, an appreciation of the poverty situation prevailing in Karak and Tafila, and the Government of Jordan's (GOJ) agricultural and rural development policies. The success of the project has in many ways just highlighted the considerable potential which still exists in the two governorates.

Effectiveness. Overall, the evaluation found that the objectives of the project had been significantly achieved. Major improvements have been recorded in soil and water conservation, agriculture, and skills levels. By upgrading and protecting the natural resource base, through technical support to farmers and by providing credit and skill development programmes (especially for women) the vulnerability of farmers, herders and rural women who have been reached by the project has been decreased. The re-allocation of targets and resources after the Mid-term Evaluation (MTE) improved the potential effectiveness of the project by better aligning targets and funds with beneficiary demands and project capacity. The only possible detracting factor has been the difficulties in better targeting the interventions to the poorest. The evaluation concluded that the performance of the project in terms of its implementation effectiveness was substantial.

Efficiency. The economic rate of return (ERR) projected at Appraisal was 10.6% based on improved cropping on private land areas which had been conserved. The evaluation considered that this under-estimated the overall ERR for the project as it ignored benefits for herders on State land and improved incomes from the women's credit programme. The evaluation therefore not only re-calculated the cropping benefits, but estimated these additional benefits as well. When all the productive activities were combined, the revised ERR was 34.8 percent. This is a very impressive result, and indicates that from GOJ's perspective borrowing for ARMP development activities was well worthwhile and should lead to increasing economic returns. Improvements to physical resources, enhanced and rejuvenated agricultural production, and promotion of off-farm income generation have also proved positive for family incomes: the evaluation found that the likely increases to farm household incomes indicated in the Appraisal (between 30 to 400%) were probably correct. Implementation efficiency of the project was therefore found to be high.

Impact of the project on rural poverty

Changes in the natural resource base have largely been positive as water storage facilities constructed by the project have resulted in improved pasture, and the area of protected and stabilised soil on private farms has increased as a result of the SWC measures and the expansion of tree cropping. Farmers either adopting SWC activities or benefiting from spring rehabilitation have seen their physical assets improved, probably best illustrated by the increased values now placed on their holdings. However, there is some concern that the project-sponsored cisterns for supplementary irrigation requirements do not adequately serve the new areas planted to olives and unintentionally the effect has probably been to add to the depletion of what is already a severely depleted aquifer.

Positive impacts on household and financial assets have also been reported by beneficiaries as a result of the Income Generating Activities (IGA) programme, and mission calculations indicate that these impacts are significant. Access to markets - an important factor in poverty reduction - has improved since the beginning of the project as a result of beneficiary (and other) demand, although this was not part of the project design. All beneficiaries have either received skills training, or have profited from increased access to information via the project staff.

As a result of high demand, credit allocations for women's IGAs, through ACC credit operations, have been significantly increased (which has also had the effect of increasing the number of women benefiting from skills training). Overall, women's access to project benefits has been good and w omen beneficiaries confirmed that they felt more empowered in the household as a result of being income earners, particularly those that had no previous employment, but benefited from ACC lending. T he credit experience has led to ACC being willing to modify its lending terms and conditions somewhat, although the changes made are small, and still do not mean that the poorest and disadvantaged will be eligible for credit.

Although the design called for participatory approaches, in practice these have not been adopted. Because of this lack of participation, the development of social capital and local self- help has not been an outcome of ARMP, neither has the project made any contributions to building local capacities for community empowerment. In addition, the evaluation concluded that beneficiaries of the SWC, farm development and credit programmes were mostly from the relatively better off bracket, therefore, although the project has had a positive livelihood on the poor in general, the poorest has not been reached as expected.

Sustainability. The IE concluded that the project SWC interventions would largely be sustainable after project closure, as the on-farm structures require little maintenance. By comparison, there are significant maintenance requirements for the mini-dams, hafiras, and check dams on State land, and it is probable that the long term maintenance of these structures will remain a government responsibility, with little possibility of cost recovery from users. Although ACC has not established a specific revolving fund it has confirmed its commitment to continue to provide loans by going beyond the initial allocations for credit. The evaluation estimated that the IGAs undertaken were in general profitable for the borrowers, and hence will probably be continued.

Innovation and Scaling up. T he deep pruning of old olives introduced a contextual innovation, though not totally unknown to Jordan . It has introduced a new technique into traditional practices and has proved to be very beneficial. This technique could be more widely demonstrated and replicated wherever it is appropriate. To an extent, the introduction of check dams was also innovative, even though this structure was not continued by the project. The project has already provided training to many women who could not take loans because they could not meet ACC's conditions. Expansion of the very successful IGA programme could take place if these women could be reached by an appropriate credit mechanism: this is an opportunity for scaling up to occur within the project area, at little additional cost.

Overall, the impact of the project on Rural Poverty was found substantial.

Performance of the partners

IFAD and the Cooperating Institution (CI). Th e evaluation concluded that whilst the intentions and relevance of the interventions could be rated very highly, there were some aspects of the design which were not satisfactory – especially the targeting criteria and the practical aspects of the implementation approach for community involvement. The arrangements for the involvement of ACC also did not offer any progress in terms of enhancing the outreach of credit to the poor. However, overall IFAD has facilitated a design which "did the right things". In terms of implementation, the evaluation agreed with the Project Management Unit's (PMU) assessment that both IFAD and the CI had generally provided good support (although in the case of the CI this was mostly in fiduciary aspects).

Government and its Agencies. The arrangements made by the Ministry of Agriculture (MOA) for project implementation have proved very satisfactory. The only difficulties experienced during implementation relating to the reluctance to change operational arrangements to meet the implementation mechanisms as specified in the loan agreement (e.g. participation) and the reluctance to use loan funds for Technical Assistance (TA) or for contracting skills through Non-Governmental Organizations (NGOs) etc. The PMU has proved a very effective mechanism for day-to-day implementation, which has been demonstrated not only by the physical achievements of the project, but by the very pragmatic way that problems have been overcome and the decisive management involved in the modification of project targets at mid-term. The ACC has been an effective partner is terms of disseminating the credit funds, but cannot be said to have contributed much to the achievement of the project's poverty objectives.

Evaluation insights

Water management

The acute scarcity of water in Jordan has been given particular emphasis in the present Country Strategic Opportunities Paper (COSOP), yet the evaluation came to the conclusion that whilst ARMP had contributed to water conservation, it had not actually focussed sufficiently on the crucial aspect of water management. Improving the livelihoods of the rural population depends largely on intensifying agricultural production and productivity, which requires optimising the use of water. Water, rather than land, was (and is) the limiting factor. To address this issue, the evaluation has made a series of recommendations (see below) covering the matching of water conservation to land development, the size of cisterns and the promotion of water user associations. For future interventions, the evaluation concluded that designs need to place much greater emphasis on water management.

Innovation in credit

As designed the credit programme in ARMP reflected a somewhat narrow view of the role of credit in rural development. In considering a second phase of ARMP, as happened in subsequent projects (e.g. Yarmuk), the approach should shift more towards the provision of rural financial services, following IFAD's policy guidelines (which had not been developed at the time of the original design of ARMP). These state clearly that to maximise impact and reach of the rural poor a wider spectrum of financial services (e.g. savings), lending modalities and partners engagement should be implemented. While ACC possibilities for being innovative in phase I to optimise financial services to the poor were limited, circumstances have now changed in that there is now much stronger GOJ and ACC's commitment to addressing the issues of poverty in the rural areas through credit and that IFAD has clear guidelines in designing its intervention in this field.

ACC has been an effective partner in disseminating credit funds in ARMP. It must continue to have a key role in any MOA and IFAD future development strategy, as it remains the only formal organisation able to provide financial services to producers in the rural areas. The need, however, is to identify the mechanisms through which the role of ACC can become more broad based, able to reach out to those individuals and groups who need most ACC services in order to enable them to overcome their poverty. Despite the achievements and successes in ARMP, the limitations demonstrated show that there is still some way to go. This will require innovation in providing financial services to the poor, and this should be a feature of the design of future interventions. Recommendations to address these aspects have been made (see below).

Women's programme

The project design included not only the women's programme, but specific references for inclusion of women-headed households in the SWC and agricultural components. Overall, therefore, women's access to project benefits has been good and approximately one half of project beneficiaries are women. The IGA programme has proved to be impressively successful in Phase I, demonstrating a useful mechanism for increasing family household incomes. Even where IGAs have not actually been started, the training provided has almost certainly been beneficial, possibly representing a further stream of benefits which the project might do well to assess. The present Women's Development Programme (WDP) provides a building block for the future, but should not just be copied. Rather, the need is to update the approach so that gender is further mainstreamed to the extent possible under local circumstances, and weaknesses identified are addressed. Recommendations have been made to address these issues (see below).

Decentralisation

The project activities have tended to substitute for the regular roles of the staff of the Field Directorates of Agriculture (FDAs). Whilst this may have come about because the project has more funding and is able to promise more to potential beneficiaries, it is not sustainable. In the second phase project activities should temporarily complement the activities of the FDAs, but a gradual hand over strategy should be designed and implemented. The evaluation concluded that consideration should be given to additional support for the local extension services, as well as a formalised handover process by which developed areas become the responsibility of the FDAs. Further, the ongoing involvement of the Regional Agricultural Coordination Committee (RACC) needs to be strengthened, so that relevant decisions about the project's day-to-day work programme can be gradually taken by the agencies most closely involved. Both these aspects would serve to reinforce the decentralised nature of the project's activities, help ensure sustainability and achieve the original intentions expressed in the design.

Recommendations Agreed by Partners

Participation

In a future second phase of ARMP the project could make a significant contribution to MOA by assisting the Ministry to develop participatory approaches in its poverty alleviation and agricultural development efforts. There is a need to adopt a strategic approach to participation and to mainstream it at all levels within the Ministry. In doing so the following steps have been agreed to incorporate in the design of the second phase:

IFAD should in future assist MOA to mainstream participatory approaches throughout the Ministry. This might include funding a series of workshops and a sensitisation campaign (similar to that conducted for establishing the Gender and Development unit), and providing TA for training trainers in the Training Department of the MOA.

• A study should be undertaken within the designated project areas that analyses the arrangements for and impacts of traditional (village development committees) and more recent community associations and identifies mechanisms that have proved effective for the mobilisation and participation of rural poor communities.
• Training should be provided for the appropriate PMU staff in community development and group formation, including study visits to other IFAD projects where community participation and organisation have been successful.
• A specialised and experienced NGO should be contracted to backstop, train and monitor the implementation of the community participatory approach.
• Education should be provided for beneficiaries in community participation, and beneficiaries should be assisted to form village development committees. Consideration should be given to the provision of community development funds.
• Steps should be taken to ensure beneficiary participation as a feature of the monitoring of project activities.

Implementation: IFAD's NENA regional division and MOA in design and implementation of phase II.

Water

Supplementary Irrigation Requirements. Most purchased water in the project area comes from private wells, all of which appear to tap in to the same aquifer (B2/A7) in the Dead Sea Basin . This aquifer is already described by the Water Authority as being "severely depleted" and any additional abstraction will accelerate this depletion: this is totally contrary to the conservation objectives of the project. A major step required to address this problem is to ensure that the capacities of the cisterns provided are adequate. Whilst this is recognised by the project, the evaluation considered that the proposals to address the shortfall were inadequate, and should be re-visited. It was therefore agreed that:

• The design principle adopted in the second phase should be to ensure that farmers' supplementary irrigation requirements can, in a normal year, be met by harvested run-off water only.
• The project to start taking steps to estimate cistern capacity based on more precise water-requirement information.

Implementation : IFAD's NENA regional division and MOA.

Water-use efficiency. The evaluation found that there was considerable variation between farmers in the frequency of irrigation and in the amounts of water applied, suggesting that both over-irrigation and under-irrigation may occur. In the context of a project in a low-rainfall area farmers, irrespective of their level of experience (and it should be borne in mind that some of the project's beneficiaries come from non-agricultural backgrounds), should have access to clear guidelines and recommendations regarding efficient water-use practices. At present, area-specific recommendations do not appear to be readily available to growers. It was therefore agreed that:

• The project should ensure that all growers are informed about and receive guidelines for supplementary irrigation practices for olives and other fruit trees, so as to ensure an optimum economic balance between yield and water use. These guidelines should be quantified for trees at different stages of maturity on the various soils in the project area, in a normal season.
• If appropriate guidelines, applicable to project circumstances, do not exist, or if they need to be refined, the project should enlist the collaboration of the National Centre for Agricultural Research and Technological Transfer (NCARTT) specifically to formulate and develop them.

Implementation: FAD's NENA regional division and MOA in the design and implementation of phase II.

Water User Associations (WUAs). An essential aspect of water management is for local community groups to take responsibility for the management of improved schemes. However, the evaluation observed considerable confusion amongst groups concerning members' responsibilities, obligations and benefits. Both to protect the investments made in construction and to ensure the proper maintenance of the schemes, there is a need to establish properly constituted WUAs. These Associations facilitate community management of a common resource and can also provide valuable "entry points" in terms of greater participation from the community. Such an approach is worth pursuing in future. It is therefore agreed that:

• More attention should be paid and resources allocated to the formation and structure of water-user groups (WUAs) on communal irrigation schemes that have benefited or will benefit from project support in the form of spring and canal rehabilitation.

Implementation : IFAD's NENA regional division and MOA in design and implementation of phase II.

Handover strategy

Jordan 's agricultural extension service is seriously under-staffed and its effectiveness is consequently compromised. This presents a problem in terms of the continuing support required by farmers in order to fully realise the development potential of improved land and water resources. At the moment the extension agents working with the project appear to have to continue their activities (after the development phase has been completed) with an ever-growing portfolio of clients. Similarly, the arrangements for maintenance of the structures developed by the project on State land are unclear, and the project seems to accept some responsibility in this respect. There is a need to address the handover process of completed conservation areas on private land for regular support by extension agents within the FDAs, and the handover of structures to Government agencies or community groups for maintenance. There is also a need to ensure that the Extension Services can cope with the additional loads after completion of the development phase. It was therefore agreed that:

• Consideration be given to formalising handover strategy and procedures for (i) completed areas developed through the project to the respective Directorates of Agriculture, and (ii) structures developed to appropriate agencies for their management and maintenance.
• Consideration should be given to providing support from project funds for the extension agents in the FDAs concerned with the project.
• Further policy dialogue between IFAD and MOA to identify the extent to which the current status of the extension service shortfall could affect the achievement of wider poverty reduction aims in the rural areas and agree on medium term measures to address this issue.

Implementation : IFAD's NENA regional division and MOA in design and implementation of phase II.

Credit

Notwithstanding the significant achievements of the credit activities under ACC management in phase I, it was agreed that in future:

• A minimum of 10-15% of credit allocations should be designated to test pilots of alternative mechanisms of group credit provision suitable to the socio-economic conditions of the rural poor. This should be administered by ACC, but channelled through appropriate local level rural organisations, for example, sanadiq if found suitable. Advantage should be taken of assessing successful Regional experience in this regard, for example in Syria . The new mechanism should be created in the project area for the specific purpose of providing loans to the most disadvantaged households who cannot meet the present collateral requirements. The arrangements should be established by the project's Credit Unit, and should operate under the joint supervision of the PMU and the ACC.
• Consideration should be given to establishing a Credit Unit within the project organisation. Such unit is needed to ensure on-going and effective coordination, as well as to endow future projects with 'credit expertise' so that they might play a more effective role with regard to credit outreach and monitoring of IGAs. If such a Unit is established the project should meet the costs of a full time officer seconded from ACC to head this Unit, plus training for all the Extension Officers within the project in assessing credit needs.
• In order to reinforce the overall participatory approach in the project, the credit aspects should, in future, include the possibility of utilising the services of a prominent NGO in the field of micro-finance to provide the skills which are not currently available⁴, such as the institutional arrangements for establishing credit groups.
• ACC should initiate means of establishing a clear and consistent relationship between the size and duration of loans on the one hand, and the purposes for which the credit is to be used on the other.

Implementation : IFAD's NENA regional division, MOA and ACC in the design and implementation of phase II.

Women's Programme

The following recommendations were agreed:

• The design of future interventions should be based on the gender mainstreaming approach which is now a core and required aspect of IFAD interventions and support, although it is recognised that this must be adapted to local cultural circumstances.
• The range of IGAs should be expanded after feasibility and marketing studies which seek to identify products that have market demand and are financially viable. These studies should be wide ranging and include small-scale group production. They will probably require some form of TA, both for the studies and to continue to backstop the activities identified.
• Future interventions should give more emphasis to capacity building for beneficiaries. This would include training in the organization of collective working, administration and marketing skills.
• Future interventions should consider funding women's centres. The necessity for women's centres is twofold: firstly, they would be more cost efficient for poor women working collectively by sharing equipment, supplies and storage space, rather than each working separately (e.g. from home where space and resources are limited for larger scale production). Secondly, women's centres would assist in institutionalising women's groups.
• An experienced local NGO should be contracted (i) to train WDP officers to apply participatory approaches methodically and (ii) to help establish women groups for small enterprises. This can be the same NGO contracted to assist with introducing participatory approaches (see under participation above).

Implementation : IFAD's NENA regional division and MOA in design and implementation of phase II.

Institutional Strengthening

An appropriate package of institutional strengthening should be included in future projects for the MOA, covering both the technical training requirements of beneficiaries and staff, and sufficient to allow for the sensitisation in, introduction of and adoption of the suggested participatory methods. In addition assurances should be sought that identified staff needs and planned training activities would be fully funded during implementation.

Implementation : IFAD's NENA regional division and MOA in design and implementation of phase II.

Decentralisation

The role of the Regional Agricultural Coordination Committee should be more clearly defined as an ongoing responsibility for the work programme of the project. This might, for example, involve the approval of all areas to be developed and quality control of the project's construction activities.

Implementation : IFAD's NENA regional division and MOA in design and implementation of phase II.

Project implementation

The project should seek to make wider use of the assistance available in other Departments of the MOA, both to address strategic issues (e.g. the marketing of olive oil) and to obtain technical support (e.g. with arranging training). In addition, care should be taken to link with the Ministry's own M&E Department, in order to harmonise reporting on the overall progress of projects.

Consideration should be given to the adoption of the M&E system devised for ARMP in a wider context: there is the potential to extend the methods used to many projects in the Region. With appropriate support ARMP could provide a local "centre of excellence" for training in M&E.

Implementation : - IFAD's NENA regional division and MOA in design and implementation of phase II.

- IFAD's NENA regional division in the design of other projects in the region.

---

1/ This agreement reflects an understanding among the key partners to adopt and implement recommendations stemming from the evaluation. The agreement was formulated in consultation with the members of the Core Learning Partnership (CLP). The latter comprised Prof. Awni Taimeh, Secretary General, Ministry of Agriculture; Mr Radi A. Tarawneh, Director ,Productive Projects Dept, Ministry of Agriculture; Mr Mahmoud Choulen, Director of Projects, Ministry of Planning; Mr Mousa Al Abbadi, Director, Dept. of Projects, Ministry of Agriculture; Mr Jihad Abu Mushrif, Director of International Cooperation; Mr Nimr El Nabulsi, Director General, Agricultural Credit Corporation (ACC); Eng. Reem Qura’n, Assistant Director General, Public and International Affaire, ACC; Mr Jamil Jaafreh, Project Director; Representative from CARE International, Jordan; Dr Mervat Badawi, Director, Technical Department, AFESD, Kuwait; Dr Abdelhamid Abdouli, Country Portfolio Manager, Near East and North Africa Region, IFAD; Dr Mona Bishay , Deputy Director, Office of Evaluation, IFAD, in charge of the evaluation. OE facilitated the process.

2/ As approved by the Executive Board, April 2003. Quote from paragraph 46.

3/ The evaluation methodology followed uses three composite evaluation criteria: project performance (composed of relevance, effectiveness and efficiency) rural poverty impact (composed of six impact domains, sustainability, innovations and gender equality) and performance of partners (including IFAD, implementing agencies, and the cooperating institution). The ratings used to assess performance using these criteria are high (4), substantial (3), modest (2) and negligible (1). For the sustainability criterion the rating used is highly likely (4), likely (3), unlikely (2) and highly unlikely (1).

4/ For an overview of NGOs in Jordan active in micro-finance see “Restructuring of Agricultural Development Banks: the Case of the ACC in Jordan ” Michael T Marx, April 2003.