The Core Learning Partnership and the Users of the Evaluation ¹

In 2004 the Office of Evaluation of IFAD conducted an Interim Evaluation of the Special Country Programme Phase II (SCP II) in Ethiopia.  An approach paper was discussed with partners in Ethiopia in May 2004, and a pre-mission socio-economic survey was fielded between June and August of the same year.  A core learning partnership was formed comprising representatives of the Ministry of Water Resources, the Ministry of Finance and Economic Development, the SCP II PCU, UNOPS and the IFAD Region for Eastern and Southern Africa (PF).  The main evaluation mission took place 13th September-14th October 2004.  A draft evaluation report was distributed in December 2004.  A final evaluation workshop was organised on 24th February 2005 in Addis Ababa (Ethiopia) to take stock of the evaluation findings and prepare this Agreement at Completion Point (ACP).  The workshop was attended by the members of the core learning partnership and other stakeholders. The ACP illustrates the stakeholders' understanding of the evaluation, findings and recommendations, their proposal to implement them, and their commitment to act upon them. 

The Main Evaluation Findings

Quantitative Achievements.  According to the UNOPS 2004 report² , 31 out of 55 small-scale irrigation schemes had been completed by April 2004.  Discussions held by the evaluation mission in the field suggest that 49 out of 58 schemes were complete by September 2004.  The water management component (introduced after MTR) is reported by UNOPS 2004 to be substantially complete, with the exception of registration and legalisation work in relation to WUAs, and several elements of local training.  The development of WUAs has been less than satisfactory because of the way in which traditional water management structures have been ignored during implementation, and because of the failure to reconcile the aims and legal status of WUAs and cooperatives.  The UNOPS 2004 report paints the general picture of very limited project achievements in the agriculture component, the most alarming single area being that of soil conservation.³  

This is an area of real concern: limited transfer of knowledge through the present extension system has prevented farmers from achieving higher yields.  In addition, as the rate of cropping intensity of land increases, it becomes crucial to compensate for the extraction of soil nutrients and prevent a decline in soil fertility and, eventually, soil erosion.  The final component, Capacity Building/Coordination, is reported by UNOPS to be well-progressed (achievement of target varying from 66% to 83% across the various sub-components).  In financial terms, up to July 2004, Regional disbursement of funds varied from 40% to 70% of totals budgeted.

Major strengths (i) relevance.  The project is relevant to the needs of farmers in traditional irrigation schemes which are located near to markets, and to those farmers who are able to benefit from expansions to these schemes.  The project has been well targeted in the sense of working mainly in woredas defined as highly or very highly vulnerable to drought and food insecurity.  The project fits well with government policies on water resources development and food security.  It is also highly relevant to IFAD's principles and strategic thrusts in East and Southern Africa.

Major strengths (ii) integrated design.  Successful irrigation does not simply consist of the application of water to land for the production of crops.  In rural Ethiopia a set of complementary activities is essential to the achievement of beneficial and sustainable impacts.  While 70% of the SCP II budget is devoted to the improvement and expansion of small-scale irrigation schemes, the remaining 30% is for soil conservation, the development of women's vegetable gardens, agricultural support services, and capacity building.  Each of these complementary activities, and those added since the initial project design (such as the strengthening of credit services and market access), is either essential to the successful performance of the irrigation schemes, or adds significantly to the overall impact and likelihood of that impact being sustained over time.

Major strengths (iii) impact.  The evaluation (see main report) provided some evidence of crop diversification, yield and production increases, and corresponding increases in agricultural income.  It also concluded that, in the limited number of cases where women's gardens have been developed, this has had a very significant impact on the women and families who have directly benefited.  In some cases physical assets other than the irrigation schemes and soil conservation works – roads - have been improved.

Major strengths (iv) sustainability.  During the evaluation many examples of the strong commitment to the project by Federal Government, the Regions and the Project Coordination Unit were evident.  This has resulted in a willingness to learn, and to modify procedures on the basis of that learning.  Particularly impressive has been the honest awareness, in some Regions more than others, of the existence of design and operation and maintenance challenges, and the willingness to find ways to solve these problems.  In the best cases, co-ownership of schemes with farmers and provision of post-construction support have more than compensated for the naivety of the scheme designers who believed that schemes could be "fully demand-led", "self-sustaining", "self-managed" and "self-directed".

Major weaknesses (i) in design.  Weaknesses in project design are set out in the evaluation report, and only four points are highlighted here since they have particular relevance to the future.  First, many assumptions have been made about the weaknesses of traditional irrigation systems, without the foundation of detailed investigation and diagnosis.  It may be that less capital-intensive interventions to improve traditional systems could spread benefits more widely.  Second, a significant number of economic assumptions made at appraisal were clearly optimistic.  In particular the high yields, high producer prices, low post-harvest losses, no water scarcity, and no maintenance costs assumed at appraisal.  It would be unfortunate if actual project performance were to be judged against measures which were themselves grossly unrealistic.  Third, little was said at appraisal about the importance of realistically assessing market potential, and selecting sites within close proximity of existing or potential markets.  Fourth, the notion, already highlighted, that modern small-scale irrigation schemes can be designed and constructed using specialist skills, equipment and materials, then handed over to farmers with no post-construction support or back-stopping, is unrealistic and unworkable.

Major weaknesses (ii) in implementation and impact.  Six areas are highlighted here.  First, the success of commercialised small-scale irrigation schemes depends crucially on access to credit and input and output markets.  Promotion of cooperative membership has only partially achieved this access, and only for some farmers.  Second, traditional water management structures have not been exploited effectively in the establishment of ‘modern' WUAs and cooperatives.  This threatens the viability of the modern structures, and is disempowering of the traditional organisations.  Third, despite some real impact of the development of women's vegetable gardens, too little of this project component has been undertaken.  Much more remains to be done in this important area of empowerment and household-level impact.  Fourth, much more attention needs to be paid to soil management issues, both within and outside the irrigation schemes.  Agricultural services in general, especially generalist extension advice, trials and demonstrations, and seed availability, need to be enhanced.  Fifth, the issue of competition, and in some cases conflict, between upstream and downstream water users is of major concern.  Sixth, the evaluation found a significant number of cases where farmers are worse off than they were without the project.  Not all ‘modern' irrigation development has benefited all of the target farmers.  Mistakes have been made in particular when engineers have ignored the knowledge or wishes of farmers, when hydrological assessments have been flawed, or where upstream developments have deprived schemes of water.

Major weaknesses (iii) in project management.  Three specific weaknesses in overall project management are emphasised here.  First, there is no agreed project logical framework or equivalent, setting out the hierarchy of project objectives and activities, together with indicators of achievement, means of verification, and risks and assumptions.  Because of this, there is no general agreement on what to monitor and how to do so.  Consequently no-one knows exactly what the project has achieved.  The supervision process has not been an appropriate vehicle for putting an agreed monitoring system in place, and nor has short-term TA.  Only a long-term substantive partnership could have solved this problem, by allowing IFAD to engage in a participative process with the key stakeholders.⁴

Second, too short a time-horizon has been taken by the project partners.  Long term commitment is needed.  A six year project is too short to achieve significant impacts.  It has taken until PY5 to reach a peak in irrigation scheme construction, and longer in the agriculture component.  Benefits to farmers will take another 6-10 years to realise.  Third, there has been too little sharing of institutional knowledge ⁵.  This applies at two levels, (i) between Regions and woredas, and (ii) between the project, Federal Government and other donors.  The first of these would allow the spread of good practice from the best-performing Regions to others, while the second would allow forward movement in policy, strategy and donor coordination.

Main weaknesses (iv) in partner performance.  The evaluation has highlighted here two important weaknesses.  First, Federal Government commitment has not always been as constructive as at present, and even now stronger adherence to the spirit and letter of agreed responsibilities would enhance project performance considerably.  Consistency and strength of commitment are needed.  Second, the evaluation expressed concerns about the institutional processes involved in the ‘partnership' between IFAD, UNOPS (the supervising organisation) and GoE.  Specifically, too little time was invested in the supervision function, and the technical assistance provided on occasions failed to develop a partnership characterised by trust, support and constructive engagement.  These weaknesses are not unique to this project and these external organisations, but they nevertheless should be taken seriously.

Recommendations Agreed upon by All Partners

Strategic Issues

• Assuming the mutual desire to continue and consolidate the successes of the project, while moderating its weaknesses, a number of key strategic issues need to be addressed.  The emphasis here is to build on the good practices and internal learning already developed, while mitigating those factors which reduce the effectiveness of the project.
• The process of internal learning was fostered through the project workshop held in Adama (Nazaret) in August 2004.  This was agreed by all to be a very positive experience.  However, the valuable and detailed sector experience which has been developed now over many years has not significantly contributed to debate and dialogue on Government policies and strategies, nor been shared widely with other donors.
• Although GoE has displayed a good deal of commitment to the project, frequent restructuring of Government has impacted negatively on project performance.  There is concern even now that transfer of responsibility for SSI from MoWR to MoA may compromise the effectiveness of SCP Phase III, at least in the short term.
• The evaluation team was concerned about the very limited time allocated to supervision from the cooperating institution (UNOPS) by IFAD.  It also criticised the TA process, by which foreign consultants advise the project and the donor, through limited time inputs and correspondingly limited in-depth dialogue with the partner.

Summary Recommendations

• Continue the start which has already been made in sharing good practice and experience between Regions and woredas, with regular workshops; include also WUAs, donors and invited guests;
• Extend the project experience more widely by developing policy dialogue with GoE and donors on topics such as water resource management at catchment level; adaptation of national water resource policies and legislation to regional level; marketing and price regulations; policies on water users' associations and irrigation cooperatives; policies and practice relating to land title; and understandings and practices in relation to post-construction maintenance and rehabilitation;
• Work to minimise the impact on the project of organisational re-structuring in Government;
• Develop a partnership between donor, cooperating institution and Government which is characterised by trust, constructive support and continuity of relationships.  In addition to external supervision the project needs continuity of constructive support from IFAD or UNOPS, together with national consultants, to assist in overcoming challenges identified by project stakeholders.

Suggested timing: immediate, and throughout future GoE involvement in small-scale irrigation development.

Partners involved: MoWR, MoARD, CPO, IFAD, UNOPS, PCU, Regions, woredas, WUAs, other donors.

The Process of Formulation of Phase III

• The emphasis here is on both the process and the outcomes of project formulation, assuming a third phase is to be developed.  The concerns of the evaluation team as they examined phase II were the domination of the formulation process by outsiders; the lack of clarity in regard to precise project aims and objectives (and consequently in regard to relevant indicators too); the inaccessibility of the outputs of the process; and, in some key respects, the unrealism of the project design.
• If the project is to move into a third phase involving construction of more SSI schemes, then its duration needs to reflect the significant time taken (a) to reach a peak of construction activity, and (b) for the full benefits of the project to be realised by the target farmers.  The impact of the project will only be fully realised if there is continuity of project activities over a minimum of 10-12 years.  Given this requirement, then the project design needs to be sufficiently flexible to allow for changes in the external environment and the learning developed by project staff..  Joint responsibility for maintenance by farmers and irrigation authorities is essential to bring about sustainability.  It is essential that the project goals, activities, indicators of achievement, means of verification of those indicators, and assumptions are set out very clearly.  A logical framework is the most convenient form of expression of these elements.  The present project includes an excessive amount of detailed documentation, which, because of its quantity is inaccessible to most project stakeholders.

Summary Recommendations

• Carry out project formulation in a fully participative manner, drawing on the lessons of experience learned by the stakeholders at all levels, and particularly those with detailed knowledge of field and farmer;
• Develop a project which is long-term, to reflect the time needed to achieve full impact, and flexible, to allow for future learning and future change;
• Develop a project which fits well with other GoE and donor projects in the small-scale irrigation sector, in recognition of the fact that the Regions and woredas are engaged in more programmes of SSI development than simply this one;
• Recognise the importance of shared responsibility for maintenance between farmers and irrigation authorities.  The distinctions between minor maintenance which can be carried out by farmers, major maintenance which is beyond the capacity of farmers but within the mandate of Regional authorities, and rehabilitation of physical and social infrastructure which requires significant external funding, need to be defined.
• Agree a full and detailed logical framework or equivalent expression of the hierarchy of goals and activities, together with indicators of achievement, means of verification, and risks and assumptions.  This should be the basis of a simple monitoring system; which should be developed prior to the implementation of Phase III.
• Work to produce clear, concise and a very limited volume of project documentation, which is accessible to all stakeholders.  All significant new project documentation should be explained and presented to relevant stakeholders in a National or Regional workshop.
•  Suggested timing: formulation process to start as soon as practicable.

Partners involved: IFAD, MoWR, MoARD, PCU, Regions, other donors

Studies and reviews of operational issues

A number of aspects of the project have been highlighted in the evaluation report as subjects which require more detailed study or review, leading to detailed recommendations for implementation or management, or acting as input to future project formulation work.  In some cases there is a wide range of experience and informed judgment within the Regions and woredas which, if brought together, can lead to sound outcomes.  In other cases some specialist external support would be of value, to supplement the experience of the National stakeholders.  In both cases, the process of debate and dialogue would benefit from assistance by a skilled facilitator.  Each of the areas of study listed below is a key area of direct relevance to project design, implementation or management.  Each study/review should be conducted in a manner which encourages the participation of all stakeholders with relevant knowledge and experience.

Catchment level planning and targeting

Although in some cases there is a real attempt to manage water in an integrated manner at the catchment level, in others water is simply taken on a first-come-first-served basis.  Conflict will inevitably increase, and it will be particularly bad in dry years.  There is relevant existing legislation, but it appears that it is not implemented.

Although most of the SSI schemes selected for development were targeted in vulnerable or food-insecure woredas, not all were sufficiently close to markets to assure their sustainability in commercial terms.  It is important that schemes are selected according to a realistic balance between need and viability.

Summary Recommendations

• Conduct participative stakeholder reviews of:
• Good practices developed within the Regions for the integrated management of catchment water resources, with a view to minimising conflict between upstream and downstream water users; this study to include review of existing legal instruments and their enforcement;
• Selection processes for SSI schemes to be improved and extended, with a view to sound targeting in relation to vulnerability and adequate proximity to markets;
• Ways in which input and output markets and market access can be developed for SSI schemes.
•  Suggested timing: as soon as possible, if SCP is to enter third phase.

Partners involved: farmers (WUAs), woredas, Regions, PCU, MoWR, CPO, IFAD, other donors 

Traditional water management associations and low-cost intervention strategy

Water management structures existed in most traditional irrigation schemes even before the project provided assistance.  Modern Water Users Associations tend to be introduced without reference to these traditional structures.  The stakeholder charged with responsibility for strengthening WUAs is only interested in promoting cooperatives.  Although cooperative membership is in principle voluntary, farmers are put in an invidious position when they choose not to join.  Although many would prefer not to join, if they do not, then they may be seriously disadvantaged, and excluded from some of the benefits which they would expect to receive as members of a Water Users' Association.  In many cases WUAs are seen by the authorities as temporary transition arrangements which should evolve into "irrigation cooperatives".  Neither WUAs nor cooperatives fully represent the water users farming within irrigation command areas.  This confusion in social organisation between traditional structures and ‘modern' WUAs and cooperatives needs urgent resolution.

More needs to be known about the weaknesses and strengths of traditional irrigation systems, in order to target assistance according to need.  It may be that lower-cost, better-targeted interventions could enable project funds to be spread more widely, with greater overall impact.

Summary Recommendations

• Conduct participative stakeholder reviews of:
• Social organisation of farmers within SSI schemes, paying due attention to traditional water management organisations, Water Users' Associations and Cooperatives;
• The performance, strengths and weaknesses of unassisted traditional irrigation schemes, in order to better focus improvements delivered by the project;
• The full range of technologies suitable for community-level farmer-managed irrigation.

• Suggested timing: as soon as possible, if SCP is to enter third phase.

Partners involved: farmers (WUAs), woredas, Regions, PCU, MoWR, CPO, IFAD, other donors

Agricultural Support and Soil Management

The present evaluation has highlighted serious limitations in two main areas: (i) ) trial and demonstration sites have been used with a limited range of crop combinations and at unrealistically high input investments; and (ii) Development Agents often have limited experience and their deployment in three specialisms (livestock, crops and natural resources) seems impractical. 

Summary Recommendations

• Conduct participative stakeholder reviews of:
• Current extension methodologies and practices, to emphasise the use of non-traditional methods, such as farmers' field visits, local fairs and competitions; the benefits of carrying out trials and demonstrations on farmers' fields should be taken into account.
• Review the curriculum of DAs, reconsider the usefulness of proposed three-pronged specialisation, review and replicate the experience of the "hirsha kadres" in Tigray
•  Suggested timing: as soon as possible, if SCP is to enter third phase.

Partners involved: farmers (WUAs), woredas, Regions, PCU, MoWR, IFAD, other donors

 Financial Issues

Concerns were expressed by many informants to the evaluation team that the cumbersome nature of some of the financial procedures creates significant obstacles to efficient project management.

It is very difficult to arrive at true capital and recurrent costs of the SSI schemes, accounting for real overheads and ‘hidden' costs ofmaintenance.  In particular the distinctions between completion costs, and costs of maintenance and rehabilitation need to be established, as do the true share of maintenance costs borne by farmers and irrigation authorities.

Summary Recommendations

• Conduct a participative stakeholder review of:
• Financial procedures at woreda, Regional, PCU, MoWR, and donor levels, with a view to simplification and acceleration without loss of accountability and transparency.  The emphasis on financial management capacity building should lie particularly at woreda level.  Regional financial accountants and others should work more closely with their woreda counterparts to develop their capacity in reporting and financial accounting.
• The true capital and recurrent costs of SSI schemes, with particular emphasis on identification of real maintenance and rehabilitation costs;

Suggested timing: as soon as possible, if SCP is to enter third phase.

Partners involved: woredas, Regions, PCU, MoWR, IFAD, UNOPS, other donors

Immediate tasks

SCP II will continue to run through 2005 (or longer if the first recommendation below is accepted).  A significant amount of work remains to be done, and it is not the intention of these recommendations to add unnecessarily to the burden of work at PCU, Region and woreda levels.  However, a minimum set of actions is necessary to achieve a satisfactory closure to SCP II.

A significant amount of the project funds remain unspent, and the evaluation team is concerned that in the haste to disburse this money the quality of both ‘hardware' and ‘software' aspects may suffer.  The evaluation team expressed concern about the limited expenditure so far on agricultural support services, including women's vegetable gardens, and encourages a re-dressing of the balance between engineering and agricultural support.

The weaknesses in project monitoring have been widely referred to.  Because of this, end-of-project reporting will be challenging.  Steps have been taken to address this problem, and the evaluation team urges the completion of this work to a high standard.  Further measures should be taken to ensure consistent and accurate reporting from the Regions in the final months of the project

Soil erosion is a major environmental threat to the irrigation schemes and in the catchments more generally, and SCP II could have a major beneficial impact in this area.  Soil erosion threatens the viability of both rainfed and irrigated farming.  SCP II includes a significant component of soil conservation work, but very little has been achieved so far.  Intensive multiple cropping in irrigation command areas will lead to soil degradation.  Without specific measures to manage soil fertility, such as rotation including legumes, and use of fertiliser and manure, soil nutrients will be rapidly depleted.

Summary Recommendations

• In order to complete the remaining project activities without undue haste and corresponding loss of quality, and in light of the accelerated AWPB in place as at February 2005, consideration should be given to requesting an extension to the project duration.
• The scheme audit which is already under way should be completed to a high standard, the data validated, and the findings collated and analysed;
• Simple consistent progress report and final report formats should be agreed between the PCU and the Regions, and implemented, with the immediate purpose of fulfilling the requirements of end-of-project reporting;
• In the final months of the project every effort should be made to re-dress the balance between activities in the engineering and agricultural support arenas.  In particular significantly more work should be carried out in relation to soil conservation, women's vegetable gardens and agricultural support services;
• The start that was made in 2004 to share organisational learning through project workshops and other relevant means should be continued.
• Specialist expertise in monitoring and evaluation (either through a retained consultant or full time specialist) should be put in place at PCU level, to facilitate the development of Regional and woreda level skills in M&E, and to coordinate overall project monitoring.

Suggested timing: immediate.

Partners involved: PCU, Regions, MoARD, MoWR, CPO, Woredas, other donors.

---

1/ This agreement reflects an understanding among the key partners to adopt and implement recommendations stemming from the evaluation. The agreement was formulated in consultation with the members of the Core Learning Partnership (CLP).  The CLP members that attended the workshop were: H.E. Mesfin Tegene, Hon. Vice Minister, Ministry of Water Resources; Mr. Adugna Jebessa, Head, Irrigation  & Drainage Development Studies Department (MoWR); Mr. Dejene Demissie, Head, International Finance and Development Institutions Division, Multilateral  Cooperation Department (Ministry of finance and Economic Development); Ms. Yeworkwha Abate (MoFED);  Mr. Ayalew Abate, Coordinator SCP II; Mr. Dele Ilebani and Mr. Robson Mutandi (UNOPS, Nairobi); and Mr. John Gicharu (CPM, IFAD/PF).  The workshop was also attended by Mr. Fabrizio Felloni, Lead Evaluator, OE and Prof. Richard C.Carter, consultant, Evaluation Mission Leader, who presented the main evaluation findings and recommendations.  A complete list of the participants is provided in the appendices to the main report.

1/In the absence of a sound monitoring system, the supervision reports constitute the best source of quantitative data on project achievements.

3/ Undoubtedly the UNOPS estimate of 6% of target achieved is an under-estimate (if only because of the omission of Amhara in these data), but discussions in the field confirm that this is an important area of under-achievement.  The fact that agricultural interventions normally follow the completion of irrigation schemes is not an adequate explanation.  Some interventions, such as soil protection, can and should be started before the construction of a scheme.

4/ Short missions, with little stakeholder participation, producing different sets of prescriptions, are of very limited value.  Long term partnership, using consultants who are trusted and respected by both partners, or through more substantial direct involvement, would have been more effective.

5/ The main exception being the Project Workshop in Adama (Nazaret) in August 2004.