Background¹

In 1996 the Programme Management Department (PMD) commissioned a Joint Review of Supervision Issues in IFAD Financed Projects. The review presented five recommendations, which were approved by IFAD's² Governing Council (GC) in 1997 together with a five year Plan of Action, which provided detailed steps and timeframes to guide implementation of these recommendations by PMD. The present evaluation covers IFAD's supervision experience under the new arrangements introduced through the 1997 Plan of Action. However, it does not contain a full assessment of the IFAD direct supervision pilot experiment because of its recent vintage.

In addition to the implementation of the Plan of Action, supervision of IFAD financed projects since 1997 has been affected by the emergence of new areas of IFAD priorities. They include: (i) more emphasis on IFAD specific aspects during implementation, i.e. participation, poverty targeting, gender sensitive implementation, participatory M&E and impact achievement and assessment; (ii) IFAD's new strategic imperatives as articulated in its Strategic Framework 2002-2006 and the efforts to strengthen IFAD's field presence and accountability; and (iii) changes in the nature of IFAD projects from input/output orientation to process orientation and community-driven approaches.

Within the above mentioned framework, the evaluation objectives were to, inter alia, assess: (i) the effectiveness of current supervision modalities against the MSRs and other indicators of quality; (ii) the current relevance of the 1998 MSRs; and (iii) the efficiency of ongoing supervision modalities.

As per IFIs practice, two main composite supervision functions can be distinguished: (i) Supervising the procurement, disbursement, end use of funds and compliance with loans/grant contracts. The supervision of these fiduciary aspects is seen as the core or mandatory function of supervision, and (ii) Providing assistance to borrowers during execution of various project activities and helping them respond to lender's requirements. This is interpreted as implementation support to the borrowers. The conventional IFIs model in supervision typically puts more emphasis on the first function and relies on distant periodic missions mounted by the financing institution.

In addition to the implementation support function embodied in project supervision by CIs, IFAD uses the term "implementation support" to denote its own activities in strengthening CIs supervision of project implementation, in order to ensure observance of its lending policies and criteria, its specific priorities and strategic imperatives. These activities consist mainly of field missions by IFAD staff and specialized consultants during project implementation for safeguarding IFAD's specificity. The evaluation reviewed this function as exercised currently by IFAD.

Overview of major findings

In its assessment of the effectiveness of supervision, the evaluation findings confirmed that the MSRs have been generally met though at a lower level than had been anticipated by the 1997 GC decision. Variations in performance were observed between the CIs and between the various tasks making up project supervision. On the whole, the largest CIs (UNOPS and the World Bank) as well as the recently started direct supervision showed a stronger supervision performance than the regional, smaller CIs. Consistently, CIs have performed better on the mandatory (core) function whereas implementation support, particularly for IFAD's specific requirements (and strategic imperatives), lagged behind. For implementation support, a function of major importance for IFAD type interventions, greater variation in performance were found between CIs. Substantial scope for improvement in implementation support by the CIs still exists, as well as for the implementation support function exercised directly by IFAD.

Good progress has been made toward a more effective portfolio management system as per the requirement of the five year Plan of Action. Such progress was achieved through the deepening of the PPR process (development of the PSR, regional portfolio reviews, and consolidated Annual Progress Report on the Project Portfolio). This enabled IFAD management and its EB to monitor more closely ongoing projects and to focus attention on critical portfolio concerns and take necessary action. Nevertheless there remains some shortcomings in the PPR process as a vehicle for improving the quality of supervision.

The evaluation highlighted the observed trend of supervision concentration (mainly in UNOPS) and the potential inherent risks associated with such concentration. This has been reinforced by the agreed phasing out of supervision by the World Bank of IFAD initiated projects. But the IFI model of supervision, also used by UNOPS, has gone largely unchanged. The clients (project managers) expressed the need for more frequent supervision when required; more participatory supervision; inclusion of more appropriate skills mix in supervision missions; better access to implementation support; higher effectiveness and relevance in support services; and more reliance on in-country or regional resources in providing such support.

Regarding the relevance of MSRs to the current institutional requirements of IFAD, the evaluation found that the standards applied for the mandatory aspects are not defined in sufficient detail in the MSRs. The definitions of implementation support are equally vague relative to IFAD's long and extensive experience in rural poverty reduction, its emerging strategic imperatives, and the aspirations of its clients. The Cooperation Agreements and Letters of Appointment for the CIs do not clearly refer to the MSRs nor IFAD's specific requirements in supervision, hence do not provide clarity and incentives for good supervision. On the efficiency of supervision, while higher supervision cost was found to be generally associated with higher supervision quality, there is scope for increasing efficiency by realistically linking supervision cost to project specific needs.

10. Overall, evaluation findings attested that the currently prevailing IFIs type supervision is not suitable for the changing nature of IFAD's support to its borrowing member countries. IFAD needs to develop different, more suitable and sustainable modalities based on building local capacities for more relevant and consistent implementation support linked to a form of IFAD's field presence.

Project supervision in IFAD is therefore faced with major challenges in the following areas:

• The changing nature of IFAD interventions requires a new supervision policy and modalities giving stronger emphasis to implementation support by local resources.
• The Minimum Supervision Requirements (MSRs) need to be transformed into an instrument that would ensure clarity and consistency in supervision of IFAD supported projects across CIs.
• The portfolio management system needs to be enhanced by adding a specific dimension to ensure the quality of supervision of CIs as well as that of IFAD.
• There is a need to reinforce the learning loop from project implementation experience and to improve the procedures for coordination between IFAD and the CIs.

Agreed upon recommendations and implementation plan

Develop a new policy of supervision in IFAD

While supervision effectiveness obtained under the present modalities is encouraging overall, there is an urgent need for improvement in the areas that respond to the changing priorities and nature of the Fund's operations. This would require changes in the present supervision practices and the development of a new supervision policy and modalities. One promising opportunity to bring about sustainable improvements in project supervision to meet IFAD's requirements relate to supporting national/regional capacity building that aim at strengthening the supply of local implementation support services to IFAD supported projects.

Following the discussion of evaluation findings with the CLP, PMD launched a revision of ongoing practices for the development of a new policy in this area. In line with its strategic framework, IFAD will favour a policy that promotes the strengthening of in-country capacities and a maximum involvement of local stakeholders in impact achievement. This approach will be reflected in the clear establishment of the distinction between:

• The mandatory (core) supervisory functions including loan administration; and
• Implementation support aimed at strengthening programme poverty impact and sustainability.

A stronger emphasis should be given to the latter to assist projects in achieving the intended poverty impact given IFAD's strategic imperatives. In formulating the new policy, this distinction will guide the definition of supervision objectives, the selection of cooperating institutions, the extent of concentration in the use of CIs, the role of IFAD and the CI in the mandatory supervision function and in implementation support, the link between implementation support and IFAD's new pilot field presence initiative and the allocation of resources to supervision and implementation support activities. A strategic emphasis on supervision quality and the strengthening of relevant standards and processes will support the revised policy.

Agreed implementation plan: While the process of formulating the new Supervision Policy has already started by PMD, its full articulation and finalisation will have to take into consideration the findings and recommendations of the ongoing Independent External Evaluation and the Evaluation of IFAD's Direct Supervision Pilot Programme by OE. The results of these two evaluations are expected by early and mid 2005 respectively. The finalization of the new policy on Supervision can therefore be expected in late 2005. This policy will be closely followed by the development of regional supervision and implementation support programmes. Time of Completion: December 2005

Revise and update the MSRs and hold the CIs firmly accountable to them

While MSRs were introduced with a view to establishing a common base for supervision carried out by all CIs and reflecting IFAD's specific requirements, this was not fully achieved. An unfinished agenda remains if the MSRs (or some equivalent common standards for supervision of IFAD supported projects) are to become the intended instrument that helps ensure consistency, quality and relevance in supervision of IFAD supported projects.

Within the framework of the new supervision policy the MSRs will need revision and reinforcement with regard to the following aspects:

• MSRs require a transformation into a system that can reassure IFAD about the inclusion of its new concerns and priorities that need to be reflected in supervision standards.
• Mandatory or core supervision functions are to be spelled out more clearly and clarification to be given as to the difference between these functions and implementation support. This would be made in reference to IFAD specific requirements and strategic imperatives.
• Incorporation of the revised requirements in the Cooperation Agreements with CIs.
• Revised letters of appointment of CIs should refer explicitly to the revised MSRs.

Agreed implementation plan: PMD to revise MSRs guided by evaluation findings and the new policy on supervision. Time of completion: March 2006 Cooperation Agreements with CIs to be revised accordingly. Time of completion: July 2006 Letter of Appointments for CIs to reflect the new required standards. Time of completion: March 2006 and beyond

3. Assess IFAD's implementation support patterns and practices

IFAD has established a system of implementation support as an input in the supervision process to strengthen CI supervision efforts; this covers a wide spectrum of technical and socio-economic inputs in support of project implementation. The evaluation, however, did not find a systematic correlation between implementation support and CI performance or with project performance. It noted, moreover, a limited use of local resources and weaknesses in the way projects interact with and benefit from implementation support experts.

It is recommended to undertake a review of implementation support practices by IFAD and their link with CI supervision and project performance. Clear principles for the use of this instrument should be established with a view to ensuring resource allocation in areas of maximum returns for project performance and impact achievement.

Agreed implementation plan: As part of the revision and updating of the MSRs, PMD will revisit its implementation support practices with the aim of clarifying its purpose, role and underlying principles; maximising project benefits from this support; and enhancing its effectiveness and relevance through the use of local resources and appropriate experts. Time of completion: March 2006

4. Improve supervision quality assurance in IFAD

The Project Portfolio Review (PPR) has become an important management tool for addressing issues identified in the portfolio. It does not at present, however, have quality of supervision as a primary objective, and its role in monitoring and improving supervision quality is minimal. Extending the PPR process to include a mechanism for rigorous review and assessment of the quality of supervision would facilitate a more consistent monitoring of CI performance, enable IFAD to give a better informed feedback to CI and form the basis for an assessment of the quality of the direct supervision carried out by IFAD.

The PPR process should be strengthened by adding a stronger institutional focus on the supervision quality assurance function. For this purpose IFAD can introduce an instrument that is designed to enhance supervision accountability for meeting well defined standards. This would apply to CIs and IFAD direct supervision. A periodic assessment of supervision quality could start out by a limited sample of the portfolio – and over the years a good part of the portfolio would be captured.

Agreed implementation plan: PMD to review the supervision quality assurance schemes in selected IFIs to identify best practices. Elaboration of a suitable supervision quality assurance scheme for IFAD (including budget). Time of completion: June 2005

5. Strengthening partnership with CIs: improving coordination and enhancing CIs performance in IFAD's specific aspects³

In assessing CI-performance, the evaluation team found that supervision of and support to IFAD specific aspects showed a generally lower performance than other supervision functions. Similar weaknesses were evident regarding IFAD's strategic imperatives. The evaluation concluded that IFAD needs to devote more consistent efforts to inform the CIs about new and emerging developments in IFAD and strengthen the knowledge flow from IFAD to the CIs. Other improvements in the collaborative procedures between IFAD and the CIs called for under the Five-Year Action Plan did not fully materialize. More frequent meetings have been held between IFAD and the major CIs but have not systematically addressed all areas delineated under the 1997 Plan of Action. Informal cooperation between CPMs and their counterparts in the CIs remains the basis for coordination and effectiveness.

IFAD should ensure that CIs get regularly informed about new approaches, strategies and policies developed by IFAD, and associated guidelines, so that they can be reflected in their supervision activities. Similarly, project staff in the field should be regularly updated in these aspects which should be included in capacity building and implementation support activities at project level. By introducing redefined supervision and implementation support requirements, areas of overlapping responsibilities between IFAD and the CI should be minimized. The practice of regular meetings in headquarters and regions should be maintained and streamlined with the aim of strengthening partnership relations and knowledge exchange.

Agreed implementation plan: 1. Setting up of mechanisms for regular briefing and communication with CIs on corporate developments in IFAD. Time of completion: April 2005 2. Reflect clear delineation of CI-responsibilities in revised Cooperation Agreements. Time of completion: July 2006

6. Strengthen the learning loop from supervision

By introducing direct supervision, IFAD was expected to, inter alia, strengthen the process of learning from its field experience, thereby closing and reinforcing the learning cycle between project implementation and design. In addition, the Plan of Action called for reinforcing the learning loop generally from supervision by IFAD as well as CIs. The evaluation found that the flow of ideas and lessons learned from implementation stays within a small group of consultants and those CPMs and CI supervisors that have responsibility for supervision. Aside from informal, personal interactions, IFAD still lacks an in-house mechanism to capture supervision based knowledge and disseminate best practices.

More needs to be done for systematizing the learning loop, from implementation to design involving the principal actors familiar with supervision on the one hand and the project formulation/appraisal on the other. This should not be limited to the direct supervision experiment but requires drawing on CI staff, as well as design and implementation consultants.

Agreed implementation plan: PMD will institute a mechanism of knowledge sharing and dissemination of learning from supervision within IFAD and between IFAD and its CIs. Time of completion: Ongoing: from 2005 onwards

7. Setting realistic fees for the CIs and base it on project needs: budgetary implications

Overall, the evaluation concluded that higher supervision performance is associated with higher supervision cost. The extensive review process with UNOPS and, to some extent, with the World Bank, revealed that IFAD's demands on CIs to undertake additional supervision work often led to very high workloads for supervision staff and is believed not to be commensurate with fee level. This has in the view of some CIs resulted in a non-sustainable working environment. On the other hand, IFAD clients have stressed the need to be flexible in supervision frequency and link it to projects needs at various implementation stages.

If the scope of supervision is to be maintained or even enhanced, including a practice of an average of two supervision missions at least in the intial phase of project implementation, a longer stay in the project area and adequate interaction with beneficiaries, realism in the setting of fees is of essence. Implementation support by IFAD is a complementary resource to the supervision work done by CIs and would have to be considered in the context of setting realistic CI fees.

Agreed implementation plan: While maintaining a zero-growth budget, IFAD is in the process of reviewing supervision requirements and resulting costs, aiming at greater flexibility in the use of existing resources, adapting supervision activities to the specific needs of the projects concerned and identifying cost-effective solutions for implementation support. Agreed upon time plan: Ongoing

Finally, it has to be emphasised that the evaluation did not examine the resource implications of the recommendations in general. Such implications will nevertheless have to be examined prior to implementation of the recommendations as the adequate availability of resources conditions the compliance with the agreed upon follow up and time plan.

---

1/ This agreement reflects an understanding among the key partners to adopt and implement recommendations stemming from the evaluation. The agreement was formulated in consultation with the members of the Core Learning Partnership (CLP). The CLP comprised IFAD's Assistant President/PMD; Director Office of Evaluation; Deputy Director Office of Evaluation; representatives from PMD; representatives from CIs; a representative from PMD Front Office; selected representatives from IFAD-supported projects; a representative from the Technical Advisory Division ; Representative from the Controller's Office/Loan. See Appendix VIII for full list of CLP members.

2/ (i) introduce and enforce minimum supervision requirements (MSRs) for the CIs; (ii) improve coordination procedures between IFAD and the CIs; (iii) establish an efficient portfolio management system; (iv) strengthen IFAD priority areas of getting results on the ground and reinforce the learning loop; and, (v) undertake an experimental direct project supervision programme by IFAD for 15 of its initiated projects (three for each regional division).

3/ This combines evaluation recommendations number 3 and 7 due to the linkages between them and their related coverage.