Corporate-level evaluation

Introduction and the core learning partnership

In 2004/05, IFAD's Office of Evaluation (OE) conducted a corporate-level evaluation (CLE) of the direct supervision pilot programme (DSPP), approved by the Governing Council in February 1997. The evaluation was undertaken in the second half of 2004 and first half of 2005. The evaluation report was finalized in mid-July 2005. A round-table evaluation workshop with representatives of IFAD management and staff, project and government authorities involved in the DSPP, IFAD cooperating institutions (CIs) and others was held in Bangkok on 28-29 July 2005 to discuss the evaluation's overall results and seek the views of the participants¹ on the draft Agreement at Completion Point (ACP). The ACP was discussed with the IFAD Evaluation Committee during the latter's 40th session on 2 September 2005. It was also considered and adopted by the IFAD Executive Board during its 85th session, held from 6-8 September 2005.

The ACP illustrates an understanding of the key evaluation findings and recommendations, proposals to implement them and a commitment to act upon them. The ACP builds on the evaluation's results and the discussions that took place during the round-table workshop. Section B of the ACP includes the main evaluation findings, and section C contains recommendations agreed by IFAD management.

The evaluation's core learning partnership (CLP)² comprised the following representatives: the IFAD Assistant President of the Programme Management Department (PMD), the Director of the Office of Evaluation (OE), one country programme manager from each of the five regional divisions in PMD, the senior portfolio manager in PMD, one senior loans officer from the Office of the Controller, and the senior evaluation officer from OE responsible for the DSPP evaluation. The CLP met on a number of occasions and provided useful comments, in particular during preparation of the approach paper, finalization of the draft evaluation report and preparation of the draft ACP.

Main evaluation findings

There is a consistent trend in the overall analysis of the evaluation which demonstrates that, compared with supervision by CIs, direct supervision has greater potential to contribute to better development effectiveness at the project level and, at the same time, allows for more attention to IFAD's broader objectives at the country programme level, such as policy dialogue and partnership building. Moreover, through direct supervision IFAD has been able to place special emphasis on issues of prime concern, such as gender mainstreaming, targeting and the building of grass-roots institutions.

Governments and other development partners at the country level have unanimously expressed deep appreciation for the more frequent contacts with country programme managers (CPMs), which has been facilitated by IFAD's direct supervision activities. The same partners conveyed that they find it more useful to deal directly with IFAD staff rather than with CI representatives. In this regard, for example, the partners conveyed that the response rate and follow-up on implementation issues are faster through direct supervision than supervision by CIs.

Direct supervision has contributed to developing IFAD's knowledgebase. In particular, the CPMs responsible for direct supervision have acquired knowledge of supervision processes, project implementation and general rural development issues in the countries concerned. This knowledge has enabled them to better design and implement new operations. However, the knowledge gained at the CPM level has not been systematically shared with others or sufficiently institutionalized, which is one of the main shortcomings of the DSPP.

The evaluation concludes that direct supervision allows the CPM to strengthen country-level coordination both within the context of IFAD operations and with the development community at large. It also facilitates the strengthening of existing IFAD-funded programmes and the identification of new programmes and cofinancing opportunities, which are mostly available at the country level, given that the majority of IFAD's international and bilateral partners have delegated an increasing amount of authority to their country representatives.

The evaluation also found that IFAD lacks a robust quality assurance system for direct supervision. As a result, the DSPP was approached and implemented in a variety of ways, based on the perception and understanding of individual CPMs. The evaluation concluded that better quality assurance would have led to even more positive results under the pilot programme.

The average cost of direct supervision per project per year (USD 93 300) is higher than the average cost of supervision by CIs (USD 61 461)³. However, the evaluation argues that costs should not be seen in isolation from the benefits that the DSPP has demonstrated. Moreover, from discussions with the United Nations Office for Project Services (UNOPS), it is clear that the amounts paid by IFAD to UNOPS for supervision need to be increased if UNOPS is to deliver the type and quality of service IFAD requires in the future. In parallel, the evaluation feels that there is potential for efficiency gains in direct supervision if, for example, the fiduciary responsibilities related to supervision are entrusted to competent national entities or greater use is made of local consultants for implementation support activities.

The evaluation revealed that not all concerned have the same understanding of the notion of supervision. There is also a lack of adequate clarity on the roles and responsibilities of IFAD, CIs, project staff and government authorities in supervision processes. Additionally, many partners at the country level felt the term "supervision" – when applied to implementation aspects of projects, rather than to the fiduciary aspects – has a paternalistic undertone and felt uneasy with its continued use.

The evaluation notes that IFAD management's attention appeared to gradually diminish following the approval of the DSPP by the Governing Council. This is illustrated by the fact that IFAD did not fully implement all the decisions of the Governing Council. For instance, a mid-term review of the DSPP was not undertaken; no integrated analytic accounting system was established to track costs of the DSPP; and no monitoring and assessment system was set up to measure the performance and impact of direct supervision. One explanation management provided for this was that the zero-growth budget policy adopted by IFAD during the DSPP period may have played an indirect but important role. According to the evaluation, however, the Executive Board also did not exercise adequate oversight to ensure that IFAD fulfilled its commitments under the DSPP. The evaluation believes that the outcome of the DSPP would have been even more significant had all the requirements laid down by the Governing Council been implemented.

Recommendations agreed upon by IFAD

The ACP includes five key recommendations, which are mutually reinforcing. They should be all implemented fully to ensure that the desired impact is achieved in IFAD's future efforts in the area of supervision and implementation support. The evaluation notes that the implementation of the recommendations below will entail new responsibilities for PMD that will require the allocation of additional staff and financial resources as well as new competencies and skills.

Recommendation 1: Develop a comprehensive supervision⁴ and implementation support policy for IFAD

The Fund should develop a specific overall supervision and implementation support policy for its operations which: (a) takes into consideration recent initiatives that are broadly related to supervision and implementation support (e.g. the Field Presence Pilot Programme); and (b) builds on the elements contained in IFAD's new operating model. The policy should include the elements described below.

Supervision of fiduciary aspects. In close collaboration with partner governments, the Fund is responsible for ensuring that the resources it provides are used for the intended purposes. In this regard, to undertake the supervision of fiduciary aspects, IFAD would – on a case-by-case basis, depending on the project or programme circumstances – decide whether to subcontract a competent national, regional or international entity to perform such functions⁵. Preference would be given to national entities, as and where reliable institutions exist, since their involvement would contribute towards building national capacities and reducing costs associated with the supervision of fiduciary functions. With regard to national entities, utmost attention would be given to ensuring that there is no conflict of interest between the entity and the IFAD operation under consideration. IFAD might itself consider to undertake the supervision of fiduciary aspects with appropriate training of CPMs. In choosing a particular alternative, professional competence, integrity and the capacity to accomplish the task in a timely fashion would constitute overriding considerations.

Implementation support. In this regard, the evaluation recommends that the policy make explicit that:

1. IFAD would be responsible for providing direct implementation support in all new projects and programmes following the approval by the Executive Board of the proposed supervision and implementation support policy. The CPMs would consequently be responsible for the process, content and outputs of direct implementation support activities. For projects that have not yet reached the mid-term review, IFAD would consider how the projects could take this policy into account⁶.
2. Such support would cover all aspects of IFAD country programmes, both at the project level and beyond. With regard to the latter, implementation support could include key aspects related to IFAD's catalytic role, such as policy dialogue, partnership-building, and knowledge management. Moreover, implementation support would pay particular attention to the human dimensions of IFAD operations, including aspects related to gender mainstreaming, participatory processes and empowerment, institution-building, and the development and functioning of monitoring and evaluation systems. Grant-financed activities under the new grants policy should also be explicitly covered.
3. The role of partner governments should be given due emphasis and specified. To this end, and where possible and required, governments will continue to be provided with capacity-building assistance to enhance their capacity to undertake more effectively the: (a) supervision of fiduciary aspects of a loan or grant agreement; and (b) ongoing monitoring of project/programme activities and providing implementation support, as required.

It is fundamental that the additional resource requirements for implementing the new policy be clearly articulated in a comprehensive and explicit manner by IFAD. This will require a detailed cost analysis, particularly of the elements described in paragraphs 14 and 15 as well as an assessment of the skills and competency of current CPMs and other concerned staff. Based on the aforementioned analysis, IFAD would develop a full proposal for meeting the additional resources involved in implementing the new policy.

The success of the new policy will also be determined by the support provided by IFAD management and the conducive environment it creates for this purpose. For example, management will need to ensure that appropriate opportunities are introduced for periodic staff training, establish an incentives framework and platforms for the sharing of knowledge acquired by CPMs, and allocate due time for reviewing experiences in implementation of the policy. The Board must also play a proactive role in exercising oversight in implementation of the new policy and in approving administrative budgets for this purpose.

The policy should be evaluable and include a roll-out and implementation plan with performance indicators that can be monitored periodically. In the interim period between now and the approval of the new policy, IFAD would continue to provide direct supervision and implementation support (as outlined in paragraphs 14 and 15) in ongoing projects/programmes in the countries included in the DSPP. Any additional projects would be approved by the Executive Board on a case-by-case basis only after legal authority has been provided by the Governing Council.

A separate section on supervision and implementation support should be included on a standing basis annually in the IFAD Portfolio Performance Report. The section should provide an analytic account of the opportunities and challenges in this area, as well as identify key lessons learned. Moreover, it should provide an indication of ongoing operational measures introduced by IFAD to address emerging issues.

The undertaking of supervision of fiduciary dimensions and implementation support will require revisiting, inter alia, Article 7, Section 2(g), of the Agreement Establishing IFAD.

Other integral aspects that the policy should consider are contained in recommendations 2 to 5.

Implementation time frame: The new supervision and implementation support policy would be presented to the Executive Board by mid-2007.

Recommendation 2: definition of supervision

The evaluation recommends that the concept of "supervision" as used by IFAD since the Report on the Joint Review of Supervision Issues and the Minimum Supervision Requirements be understood from now onwards as consisting of two distinct yet operationally linked components:

1. supervision of fiduciary aspects; and
2. supporting programme and project implementation⁷.

The function related to supporting programme and project implementation might include, inter alia, undertaking the periodic implementation support missions (previously called supervision missions), providing guidance for the preparation and implementation of annual work plans and budgets, following up on the recommendations of implementation support activities, identifying implementation problems based on interactions and dialogue with project authorities and other partners at the country level, providing timely support in problem-solving throughout project implementation both during and outside missions, facilitating access to knowledge and information to improve implementation, and organizing occasional ad hoc missions to provide technical assistance to projects or programmes, as required. This would also include oversight of project and programme implementation, for example, in terms of monitoring the achievement of physical targets⁸.

Implementation time frame: This definition would be utilized within the framework of the new comprehensive supervision and implementation support policy for IFAD.

Recommendation 3: supervision and implementation support in the framework of the COSOP

The evaluation recommends that an overall approach to supervision and implementation support be developed at the time of preparing the country strategic opportunities papers (COSOPs)⁹. This would take into account the need to supervise the fiduciary dimensions of all operations (as per the proposal in recommendation 1), and the provision of implementation support to the country programme, in addition to the traditional support provided to projects so far. More specifically, supervision of fiduciary dimensions and implementation support should be seen and planned in a holistic manner with the country programme at the centre. The country-level strategic thrusts, where applicable, will be supplemented by project-specific requirements. These will be articulated in the project formulation reports.

The DSPP evaluation report includes other specific recommendations on this topic. IFAD concurs with all these recommendations and plans to implement them. However, it also feels that some of the recommendations are too detailed for inclusion in the ACP, which should capture mainly the broad directions for future action by IFAD.

Implementation time frame: The recommendation should be implemented in all COSOPs that are prepared after the approval by the Executive Board of the new supervision and implementation support policy. The key responsibility for implementation of this recommendation will remain with each regional division in PMD.

Recommendation 4: quality assurance system

Within the framework of an overall, enhanced quality assurance system at IFAD, there is a need to significantly improve quality assurance mechanisms for the supervision of fiduciary dimensions and implementation support activities. The evaluation recommends that:

1. IFAD should establish a management review committee within PMD at the departmental level devoted to reviewing supervision and implementation support activities, results and related operational issues. The committee would meet at least twice a year for this purpose;
2. Six-monthly reviews of supervision and implementation support activities should be undertaken at the regional division level within PMD. The summary of discussions at these meetings should be circulated to all PMD divisions; and
3. Building on the experience of other international financial institutions and United Nations and other development organizations, an IFAD-specific quality assurance system should be established, which would review aspects of supervision and implementation support. The Fund would need to thoroughly reflect upon the most appropriate structure for such a group, which would ensure the most objective and independent review possible of its supervision and implementation support efforts¹⁰. The introduction of such a quality assurance group should take into consideration the mandates and performance of existing quality control systems within IFAD, such as the project development teams, the Technical Review Committee, and the Operational Strategy and Policy Guidance Committee.

Implementation time frame: PMD would implement recommendations 4(i) and 4(ii) by December 2006 and report to the Board on their progress in April 2007. Recommendation 4(iii) would be implemented with the framework of the Action Plan: IFAD Management's Response to the IEE by September 2007.

Recommendation 5: learning and knowledge management

Necessary incentives need to be provided to IFAD staff, partners and beneficiaries to share the knowledge they acquire. For example, time needs to be carved out in the CPMs work load to accord higher priority to sharing knowledge generated through supervision and implementation support. Documenting and sharing knowledge should be included as an indicator in assessing the annual performance of CPMs. These recommendations would be implemented from the start of 2007.

Specific instruments need to be established for facilitating learning and knowledge-sharing inside and outside of IFAD. In particular, time should be reserved on a standing basis in the CPM forum for discussing issues and sharing knowledge generated through supervision and implementation support activities. Each project mid-term review and project completion report should include a specific treatment of supervision and implementation support issues, as should all evaluations undertaken by OE. Summaries of all OE evaluations should be posted on the Evaluation Knowledge System web site. The project and country status reports should be reformatted to include a narrative section on supervision and implementation support. Other instruments, such as peer reviews at the PMD divisional level in relation to implementation support activities, would also be introduced.

Monitoring and evaluation systems at the project level need significant strengthening, if they are to contribute effectively to learning and knowledge generation by IFAD, its partners and beneficiaries. Moreover, in line with the new operating model, it is necessary to assist in developing integrated monitoring and evaluation systems at the country level. Every new COSOP developed starting from 2007 would include a proposal for setting up a monitoring and evaluation system at the country level, including objectives, resource allocation and operational modalities. IFAD would periodically organize workshops at the country and regional levels as mechanisms to cross-fertilize experiences and share knowledge.

Finally, IFAD would build on the experiences of other international financial institutions and United Nations and other development organizations and make more comprehensive use of information technology for knowledge management purposes in relation to supervision and implementation support. In this regard, it is recommended to expand the existing Programme and Project Management System (PPMS) so that it can carry updated summaries of supervision and implementation support activities at all times. An enhanced PPMS, when ready, would be made accessible to external partners through the IFAD web site.

Implementation time frame: The time frames for the implementation of each of the above recommendations are included in the corresponding paragraphs. PMD and its regional divisions are responsible for their implementation.

 

 

 

 

 

 

---

1/ This included directors/coordinators of the projects included in the DSPP, government representatives from the countries in which IFAD undertook direct project supervision, representatives of IFAD management, country programme managers and other staff, officials from various international development organizations, IFAD cooperating institutions and others.

2/ Members of the CLP included the following IFAD staff: Mr Nigel Brett, Mr Jim Carruthers, Mr Pablo Glikman, Mr Shyam Khadka, Mr Luciano Lavizzari, Mr John McGhie, Mr Ashwani Muthoo, Ms Rasha Omar, Mr Mohamed Tounessi and Mr Joseph Yayock.

3/ The actual amounts paid at present to UNOPS is around USD 79 000. It is worthwhile to note this, as UNOPS is serving as CI in a large majority of IFAD-funded projects. Hence, it may be appropriate to use the UNOPS supervision costs as the benchmark for assessing direct supervision costs. Moreover, it should be noted that IFAD conducted on average two supervision missions per project per year in the context of the DSPP, as compared with one mission by CIs.

4/ The term "supervision" from now onwards in this document means "supervision of the fiduciary aspects" related to IFAD financing. See recommendation 2 for the definition of supervision.

5/ In this regard, it will be important to establish clear criteria to facilitate the decisions of management and also define the means to verify that potential partners actually meet the required standards and criteria.

6/ The evaluation recommended that IFAD should undertake implementation support in all its operations globally. However, management felt that there would be limited value added in undertaking direct implementation support in projects and programmes that are already effective and have only a few years left of implementation before closing. Moreover, the approach outlined in paragraph 15(i) would reduce the administrative burden to the Fund in arranging the transfer of all ongoing projects and programmes entrusted to CIs to IFAD for direct implementation support. Notwithstanding the foregoing, it is useful to note that the same evaluation recommendation states that: "Until the required level of financial resources are made available to the Fund, additional staff recruited in PMD and their competencies and skills developed and upgraded, IFAD may consider a phased approach to expand implementation support in all operations."

7/ This could also be called "implementation support".

8/ While the CPMs would be responsible for the overall output of such activities as outlined in paragraph 24, the activities may be undertaken in a variety of modalities including, for example, directly by the CPM or by making use of the services of individual consultants or local institutions.

9/ In fact, the COSOP could include a section devoted to supervision and implementation support matters.

10/ Although the ACP includes slightly different language in this paragraph as compared with the evaluation report, there is no deviation from the essence and fundamentals of the concerned recommendation arrived at by the evaluation.

---

Phto caption 1: International Roundtable Workshop held in Bangkok, Thailand, 28-29 July 2005. From Left: Mr Luciano Lavizzari, Director, OE, Mr Jim - Carruthers, Assistant President, PMD, Mr Ashwani Muthoo, Senior Evaluation Officer, OE. - Photo provided by local photogrape