Extract at Agreement at Completion Point 

Completion Evaluation¹

The Core Learning Partnership

IFAD's Evaluation Policy² defines the objectives of the Agreement at Completion Point (ACP) as… "(i) to clarify and deepen the understanding of evaluation recommendations, document those that are found acceptable and feasible and those that are not; and (ii) flag evaluation insights and learning hypotheses for further future discussion." The role of the Office of Evaluation is to facilitate this process, within the Core Learning Partnership (CLP), leading to the conclusion of such an agreement. Because the Smallholder Livestock Rehabilitation Project (SLRP) has closed, the ACP is concerned to identify lessons from the implementation experience which are either applicable to other ongoing projects in Lebanon or elsewhere in the region and to country strategies and programmes. Members of the CLP include representatives from the Ministry of Agriculture, project and IFAD staff, representatives from AFESD and OPEC.

The main evaluation findings³

Performance of the project

Relevance. Overall, the evaluation found that project objectives were relevant to the Government's development policy and smallholders needs at design, and many remain still relevant at the time of evaluation. However, the relevance of some objectives has declined over time. The project objective of restoring livestock support services was found to be very valid at design and remains valid at the time of evaluation. The objective of establishing an institutionalised credit mechanism for small and poor rural households remains relevant. A number of factors contributed to the decrease in relevance of the objective of replenishing the livestock resources of the Bekaa with respect to the intended target group. These include the slow start to the project, the late arrival and distribution of animals, and the cancellation of the small ruminants programme. Institutional strengthening for the Ministry of Agriculture (MOA) was identified in the design as a vital objective for implementation, on the basis that services had been greatly disrupted by the civil war. This has now been achieved and continued support for this objective is no longer necessary.

Effectiveness. Effectiveness is assessed as the extent to which the objectives of the project have been achieved. Despite the limited percentage of women oriented activities in project cost the evaluation found that the project had been particularly effective in enhancing the livelihoods of women beneficiaries, directly by improving incomes and upgrading skills and indirectly in terms of gender equity and empowerment. The project has also been effective in improving access to information and knowledge for all project beneficiaries, and has partially fostered the development of institutions for the rural poor through the formation of cooperatives for rural women, which are accessible to men as well. However, the cancellation of the small ruminant activities, as well as uncertainty about the extent to which some project benefits have reached the target group has meant that the restocking objectives, as foreseen at design, have not been achieved to a significant extent, despite being a major project focus.

Efficiency. Overall, it was concluded that the incremental benefits for the dairy industry in the Bekaa from the introduction of the dairy herd under SLRP were likely to be commensurate to the costs incurred by the project in providing these benefits. However, evidence suggests that the introduced technology had not been appropriate for many of the small-scale farmers who were the initial target group of the project, rather it had been appropriate to large scale, high productivity farmers.

Impact of the project on rural poverty

The evaluation concluded that the project had resulted in substantial impacts on the Physical and Financial Assets for successful medium-scale dairy farmers, and for most of the rural women who had taken loans. Ho wever, for the smaller and/or poorer farmers, who generally considered that dairying was high risk, the evaluation could not establish any impact or reduction in vulnerability. The evaluation found that the project had provided a significant amount of training for all beneficiaries, and that this had had a useful effect in terms of increasing levels of knowledge and skills, leading to the possibility of improving living standards in the longer term. It was further concluded that the project had somewhat improved household food security in the project area, either through milk produced being kept for home consumption, or by generating additional income. The Institutional Strengthening provided for the MOA had also significantly enhanced the capabilities of Ministry staff.

The evaluation found two contrary experiences in Social Capital development in SLRP. Through the Rural Women's Programme over a thousand women have become members of women's cooperatives and have described improvements in their social standing and empowerment: this is considered a major impact of SLRP. The successful experience of the formation of the first women's cooperatives in the Bekaa has also demonstrated the usefulness of cooperatives as self-help organisations. In contrast, small dairy farmers did not experience any substantial strengthening in their local level institutions and empowerment, rather they realised their weakness in the face of milk collection cooperatives which they were unable to influence. In assessing impacts on social capital, it must be remembered that these were not specific objectives of the project design, rather unintended results.

Sustainability. The circumstances of small dairy producers have changed since appraisal and these enterprises may now be marginal or unprofitable at the current cost of concentrates and milk prices. The evaluation considered that the present arrangements for the Milk Collection Centres (MCCs) are unsustainable. Some of the livestock support services which can be offered through the private sector are likely to prove sustainable providing they are not undercut by competition with government. Although the evaluation concluded that many of the Income Generating Activities (IGAs) financed are likely to be sustainable, there is a need to enhance the overall sustainability of benefits through the establishment of an autonomous self-reliant mechanism that can perpetuate the provision of rural financial services to the poor in the Bekaa. The substantial entrepreneurial and social development benefits received by rural women through Rural Women Programme (RWP) are likely to be sustainable.

Gender. Women's social status has effectively been enhanced as a result of the success of the Rural Women (RW) cooperatives. More women are now able to work outside their homes, they became active members of cooperatives, have demonstrated their self-reliance, and as income earners, their decision-making role in the household has been strengthened.

Innovation. The model of women's self help cooperatives developed by the Rural Women Unit (RWU) was very innovative, and is probably replicable in other parts of Lebanon . There is some evidence that the YMCA may have already replicated this model elsewhere. The project introduced AI to the Bekaaa, which can also be regarded as a contextual innovation to improve genetic potential. Potential for replication and scaling up exists particularly if it is provided by competent and adequately motivated operators. This service can be sustained through the private sector. The evaluation found, however, that Artificial Insemination (AI) could not substitute for bulls in the more remote rural areas, as intended.

Performance of the partners

IFAD, the Cooperating Institution (CI) and the Co-financier. Overall, the evaluation concluded that IFAD and the CI have performed reasonably well in facilitating a design that was appropriate to the circumstances at the time, but had performed modestly in providing guidance to the project in making strategic decisions during implementation. Project staff reported favourably on support received from the CI, whose reports have mostly indicated a very satisfactory implementation situation. However, the evaluation concluded that this was overly optimistic with respect to assessments of development impacts, expected benefits and beneficiary participation. The OPEC fund co-financed the livestock credit activities; project sources reported that OPEC had been an active partner in the project. The OPEC Fund as well IFAD and the CIs had been flexible in allowing re-allocation of project funds among various activities.

Government and its Agencies (including project management). MOA facilitated the granting of autonomy to the Project Management Unit (PMU) and has supported the project with seconded staff and counterpart funds. However, delays in procuring the animals have reduced the relevance of the intervention for the small holders in the critical post conflict situation, as well as impact on rural poverty. Although research programmes were started at the Lebanese Agricultural Research Institute (LARI), and considerable investment in reconstruction took place, the programmes were only continued as long as SLRP provided the funding. As soon as they became the responsibility of LARI, they were stopped. For most of the project life the project management has effectively pursued implementation in a dynamic and technically competent fashion, and has successfully implemented the training activities for staff and beneficiaries and the programme for rural women. However, the evaluation was concerned at some of the decisions reached during implementation, particularly those concerning aspects of project design. These have affected the direction and sustainability of the project, the extent of outreach to the target groups and the impacts achieved for poorer beneficiaries.

Evaluation insights

Creating social cohesion

The women's cooperatives, formed as a means of bringing women together for their mutual financial advantage, have led to the identification of common concerns and interests, thus diffusing socio-economic barriers and building social capital. Given the difficulties created by 17 years of civil war, the promotion of social cohesion is considered very relevant, and is a major, positive effect of project activities. The social benefits have been derived through the development of the collective work, equal partnership and joint risk taking; these have drawn the group members together, facilitating the exchange of old allegiances with new. The formation of these women cooperatives has demonstrated a mechanism for realising social and economic development in poor rural villages. This was not the case for milk collection cooperatives.

Cooperatives as development partners

There have been two opposite experiences in SLRP with cooperatives. On the one hand the success of the women's cooperatives has illustrated just how useful and beneficial properly constituted and run cooperatives can be in the development effort, whilst the experience with the cooperatives for milk collection, which had mostly withdrawn their services at the time of the evaluation, has illustrated some of the difficulties. The difference relates to the ways the cooperatives were selected and constituted, and illustrates the importance not to assume that all cooperatives will be effective development partners. In the case of the women's cooperatives the groundwork for the formation of women's groups was carefully laid and was based on the success of the individual credit loans. The formation of cooperatives was then just a further but logical step to protect and enhance the gains made. The members knew they were coming together for a common purpose. By comparison, for milk cooperatives the evaluation found that there were often very general aims for cooperation, little cohesion in the membership, little clarity on potential for achieving mutual benefits and little training to this effect. Hence, the careful selection and contribution of cooperatives is a vital issue for development interventions.

Planning a role for the private sector

The project successfully contributed to the development of livestock support services, which have led to the rapid development of the dairy industry throughout the Valley. The evaluation concluded that the combination of the improved veterinary, vaccination, AI and fodder services and demonstrations had played an important role overall and touched on the activities of many farmers. Limited outreach has been partly caused by the lack of a communications component in the project. However, with limited funding available to the MOA, an important element in the continuing support of the livestock sector is the encouragement of private operators to take an increasing role. Whilst the project may have stimulated private sector services for livestock support, at the moment these services have to compete with subsidised government services. Although the government may wish to protect small scale operators from potential over charging of services, the evaluation concluded that there are at present mixed messages in the industry which are leading to uncertainty and hindering development of private sector services. The lesson to be taken from this experience for future projects is the need to plan an exit strategy for government agencies after services have been developed, to encourage a larger role for the private sector and enhance sustainability.

Recommendations agreed upon by partners

Cooperatives. In assessing the experience with cooperatives, the evaluation concluded that there were specific lessons which could be generally applicable to future interventions. Hence the agreed recommendations given below should be used as a basis in future project design and implementation in similar situations in the Neat East and North Africa (NENA) region:

• Where cooperatives are to be utilised in future IFAD-supported projects, great care needs to be exercised in their selection, effective management, and equity in the distribution of benefits to their members. When cooperatives are required to provide specific services, for example milk collection, they should be established for this purpose only, and their charters should fully reflect the interests of their members, most of whom should be small milk producers. Membership also needs to be based on contributory shares, so that each member has a financial interest in the cooperatives' success. In addition, such cooperatives need to have sufficient working capital so as to pay their producers on time, and they need exclusive collection areas, allocated on the basis of milk production density. The starting point to address the problem of marketing for small-scale milk producers is to promote the establishment of reliable milk collection cooperatives: only then may MCCs have a role.
• Whilst the women's self-help cooperatives have demonstrated their ability to promote development activities and gender equality, they are still in the early stages of their development and require longer term assistance to ensure they can "graduate" to being fully independent. Lessons from this successful experience should be replicated where appropriate in the NENA region, and incorporated in respective IFAD country strategies.

Private Sector Initiatives . With limited funding available to the MOA, an important element in the development of the livestock sector is the encouragement of private operators to take an increasing role. This is an important plank of the Government of the Republic of Lebanon (GRL) policy and in other countries in the region. Following project closure, the delivery of AI services has been adopted and is being continued by veterinarians and technicians working in the private sector. Vaccination and animal health services are similarly available, but in both cases have to compete with subsidised government services. The agreed lessons learnt from this experience are:

• If the government wishes to encourage private sector activities in the provision of AI, then it needs to limit its role to regulating the service and ensuring that technicians are properly trained and work to industry standards. Equally, for vaccinations, Government should regulate and control vaccine quality and type, and identify and restrict itself to supplying only public-good vaccinations.
• More generally, where project designs establish or expand Government services, exit strategies need to be considered to ensure that these services will not result in ongoing financial burdens for Government, and to ascertain sustainability.

Credit mechanisms. The design of the credit interventions in SLRP was in accord with the credit approaches of the early 1990s, i.e. was concerned with 'opportunities provided', rather than as an 'approach to self-development' and sustainable rural financial services. This needed to be gradually introduced and nurtured. The shortcomings of the earlier approach in terms of development impacts and, particularly, sustainability have emerged from the implementation experience. It is therefore agreed that:

• in future projects credit needs of smallholders should be considered in the wider view of rural financial services, following IFAD's policy guidelines (which had not been developed a the time of the original design). Rather than place the emphasis on how quickly loans are provided and, ultimately, on how many loans are disbursed, emphasis should be placed on creating a system which - regardless of how gradual or how modest in terms of absolute loan numbers - has the potential to grow incrementally and to be self perpetuating on the basis of its generated resources. The mechanism also needs to be managed by the beneficiaries, so that they have the major responsibility for loan usage and repayments.

Size and purpose of loans . The credit experience also highlighted the importance of flexibility when it comes to how loans are to be utilised. Loans provided to women were not linked to a specific type of economic activity and three years after exposure to USD 500 loans, women were ready for larger credit opportunities, provided through the medium of cooperatives. By comparison, larger livestock loans were provided for very specific purposes and were quite inflexible. For livestock, it was assumed that members of the target group, who may never have taken loans before, could successfully manage relatively larger credit linked to specific economic activities, whilst women, it was assumed, would be better served by smaller loans that could be used to finance various types of IGAs. The latter approach has proved more effective as well as being more consistent with reaching the intended target groups. It is therefore agreed that:

• In future, design and implementation of credit systems need to avoid enforcing a particular economic activity. Project beneficiaries should have the freedom (within limits) to embark on economic activities that are consistent with market demand.

Monitoring and evaluation. During implementation of SLRP the project management has modified the original project proposals, with IFAD and CI's approval, in major ways that have influenced project ability to achieve design goals and objectives. This was partly allowed for in the design arrangements, as data were to be collected during the first years of implementation to refine targets, and studies were to be undertaken to determine some of the implementation aspects. Whilst this flexibility is now sought in IFAD projects, and could have made the project objectives and activities more relevant to the intended beneficiaries, in practice this did not occur. The main reasons have been an absence of an assessment of the coherence of these changes with, and its consequences on project objectives, and limited beneficiary participation and assessment during implementation. No mechanisms were developed to ensure feedback from beneficiaries on project services and assessment of project impacts. An effective feedback and impact assessment system should have been the means through which important strategic decisions are made both by the project management and the Project Coordinating Committee (PCC) to ensure reaching the intended target group. The lesson from this experience stresses the crucial role that a participatory M&E system should play to build flexibility into project design and ensure at the same time achievement of the intended objectives and target group reach. It is therefore agreed that:

• The more flexibility is to be built into the design, the more emphasis needs to be given to ensuring the establishment of an effective participatory M&E system, including adequate beneficiary assessment and feedback, as well as impact monitoring and assessment.
• Changes in design during implementation need to be associated with careful assessment of their coherence with project objectives and poverty targeting.

Interventions in post-conflict situations. The basic rationale for re-stocking was to restore lost production and provide the means for poor farmers to earn a living. In the case of SLRP, although the project was designed in 1991, the first consignment of heifers did not arrive until late in 1996; three years after effectiveness and more than five years after project design. Five years was too long to wait. Inevitably, many livestock farmers would by then either have moved on, or found other means of replacing their lost stock. By 1996 restocking was possibly not what was required anyway, as it interfered with the commercial development then taking place in the dairy industry of the Bekaa. In addition, the heifers were only available as part of a credit package which, for the intended beneficiaries at project design, was far too large in comparison to their incomes, hence discouraging them to acquire it. Whilst these arrangements may be appropriate in some development circumstances, they were hardly conducive to the prevailing post conflict situation in Lebanon . The requirement was for very speedy action to restore production, followed at a later date by considerations of re-couping some of the costs involved.

• Future IFAD and partners interventions in post-conflict situations should be speedily implemented to restore production and should be followed by consideration for enhancing continuous developmental impact and sustainability.

---

1/ This agreement reflects an understanding among the key partners to adopt and implement recommendations stemming from the evaluation. The agreement was formulated among the following: Mr L. Lahoud, Director General, Ministry of Agriculture (MOA); Dr Mohamad Farran, Advisor to the Minister, MOA; Dr Joseph Tarabieh, Director of Cooperatives Department, MOA; Mr Ali Raad Abdullah, Project Director SLRP; Ms Neila Choueri, Women Programme; Mr Darwish Noureddine, Head of M&E, SLRP; Dr Mona Bishay, Deputy Director, IFAD's Office of Evaluation, in charge of the evaluation; Mr A. Abdouli, Country Programme Manager for Lebanon. OE facilitated the process.

2/ As approved by the Executive Board, April 2003. Quote from paragraph 46.

3/ The evaluation methodology followed uses three composite evaluation criteria: project performance (composed of relevance, effectiveness and efficiency) rural poverty impact (composed of six impact domains, sustainability, innovations and gender equality) and performance of partners (including IFAD, implementing agencies, and the cooperating institution). The ratings used to assess performance using these criteria are high (4), substantial (3), modest (2) and negligible (1). For the sustainability criterion the rating used is highly likely (4), likely (3), unlikely (2) and highly unlikely (1).