Exexutive summary

Introduction¹

Improving the quality of project implementation and achieving better results on the ground has been a priority for the International Fund for Agricultural Development (IFAD) since the early 1990s. This gave rise to a series of studies to examine and clarify the role of project supervision in the IFAD project cycle². In 1996 the Programme Management Department (PMD) commissioned a Joint Review of Supervision Issues in IFAD Financed Projects in collaboration with four major Cooperating Institutions (CIs). The review presented five recommendations: (i) introduce and enforce Minimum Supervision Requirements (MSRs) for the CIs; (ii) improve coordination procedures between IFAD and the CIs; (iii) establish an efficient portfolio management system; (iv) strengthen IFAD priority areas of getting results on the ground and reinforce the learning loop; and, (v) undertake an experimental direct project supervision programme by IFAD for 15 of its initiated projects (three for each regional division).

The Governing Council (GC) approved the recommendations of the Report of the Joint Review on Supervision Issues and an associated five year Plan of Action (GC 20/Resolutions, 21 February 1997, 102XX). The five year Plan of Action (1997-2001) provided detailed steps and timeframes to guide implementation of these recommendations by PMD. As the Plan of Action was to be completed by December 2001, the Office of Evaluation (OE), in agreement with PMD, decided to undertake an Evaluation of the Supervision Modalities in IFAD projects.

In addition to the implementation of the Plan of Action, supervision of IFAD financed projects since 1997 has been affected by the emergence of new areas of IFAD priorities. These have been fully taken into consideration by the evaluation, they include: (i) more emphasis on IFAD specific aspects, i.e. participation, targeting, gender mainstreaming and gender sensitive implementation, and stressing impact achievement and assessment during implementation; (ii) the emergence IFAD's new strategic imperatives as articulated in its Strategic Framework 2002-2006 and the efforts to strengthen IFAD's field presence and accountability; and (iii) changes in the nature of IFAD projects from input/output orientation to process orientation and community-driven approaches.

The evaluation objectives were to assess: (i) the effectiveness of current supervision modalities against the MSRs and others indicators of quality and review the achievements under the Five-Year Plan of Action; (ii) the adequacy of current supervision modalities seen from different partners perspectives; (iii) the efficiency of current supervision modalities; and, (iv) the current relevance of the 1998 MSRs. As most of the directly supervised projects are in the early stages of implementation, the evaluation only reviewed emerging characteristics of the direct supervision programme, including an initial analysis of the cost structure.

The evaluation methodology consisted of: (i) a desk review (DR) and assessment of supervision reports in 57 representative projects by regions and CIs; (ii) field partners' perspectives from a survey of 112 project management units (PMUs) (response rate 60%) cross-referenced through field visits to selected ongoing projects; (iii) IFAD staff perspectives from questionnaires sent to 65 Country Portfolio Managers (CPMs), regional directors and other operational staff (63% response rate), direct discussion with IFAD staff and PMD's assessment of project supervision in the project status reports (PSRs); (iv) perspectives of managers and staff of the two main CIs – the United Nations Office for Project Services (UNOPS) and the World Bank; and, (v) analysis of supervision costs collated by the IFAD Office of Internal Audit (OA).

Supervision of IFAD projects

Four supervision functions have been defined for IFAD (as per International Financial Institutions (IFIs) practice) and elaborated in the 1996 Report of the Joint Review (Box 1). Because of their open-endedness, the third and fourth functions (referred to jointly as implementation support by the supervising entity) have a large resource absorption potential. In addition as part of the 1997 Action Plan IFAD developed in 1998 the MSRs for its CIs (Table 1 of the Annex).

Supervision Functions Supervising the procurement, disbursement and end use of funds (mandatory, fiduciary or core) Monitoring compliance with loans/grant contracts (mandatory, fiduciary or core) Facilitating implementation by helping borrower's interpret and respond to the lender's requirements (discretionary) Providing substantive implementation assistance to borrowers (discretionary)

Supervision during project implementation, despite its importance, is only one of many factors that determine project performance. These factors include: (i) quality of the project design and preparation; (ii) quality of project management; (iii) the political and economic environment; (iv) commitment and ownership by government, co-financiers and implementation agencies; and, (v) quality and timing of supervision inputs.

Since the GC decision of 1997 IFAD has reduced the number of CIs used by one (Inter American Development Bank) while the share of the African Development Bank (AfDB) and the Asian Development Bank (AsDB) in supervising IFAD projects declined markedly. The five main CIs (UNOPS, World Bank, Corporation Andina de Fomento (CAF), the Arab Fund for Economic and Social Development (AFESD) and Banque Ouest Africaine de Developpement (BOAD) have increased their share of supervising IFAD projects (from 80% to 85%) between 1996 and 2002. These CIs, with IFAD direct supervision, were responsible for supervising 92% of the portfolio in December 2002. (Table 2 of the Annex). The evaluation concentrated its analysis on these five CIs and to the extent possible IFAD direct supervision.

In addition to the implementation support function embodied in project supervision by CIs (paragraph 6), IFAD uses the term "implementation support" (sometimes also referred to as implementation follow up by IFAD) to pursue its policy stated in the Lending Policies and Criteria³ (1994 version, paragraph 43): "The Fund... will itself actively participate in [the supervision of project implementation], in order to ensure observance of its lending policies and criteria". This activity/ies consists mainly of missions by IFAD staff and specialized consultants to visit projects for safeguarding IFAD's specificity and to supplement and strengthen the supervision by CIs. IFAD staff have access to various sources of funds to supplement CIs work in the implementation support area. The evaluation briefly reviewed this function as exercised currently by IFAD.

Effectiveness of current superivison modalities

Desk teview of supervision quality

Quality of supervision. The evaluation considered four elements as indications of the quality of supervision (i.e., its effectiveness) based on expectations of the 1997 GC decision as well as IFAD's priorities: (i) adequacy of supervision inputs and processes; (ii) how well CIs are meeting the MSRs; (iii) the degree to which IFAD specific aspects are being addressed through the supervision process; and (iv) the degree to which IFAD's strategic imperatives are being incorporated into the supervision process.

Supervision inputs. These were defined as frequency and length of supervision missions (SMs), size and composition of missions and mission leaders turnover. The evaluation found that:

• most CIs carry out one mission per year, the World Bank and IFAD has closer to two missions per year;
• IFAD direct supervision and the World Bank have the longest average length of SM (18 days) and AFESD the shortest (five days). AFESD and CAF mission durations did not allow adequate time for field visits.
• CIs have had various levels of variations of SM leadership with UNOPS having the highest turnover. Evaluation findings indicate that the lack of continuity in supervision mission leadership is reducing the effectiveness of supervision and project performance.
• the World Bank and IFAD direct supervision, on average, have the largest SMs followed by UNOPS then the other CIs. Supervision findings indicate that the larger SM team, allowing various disciplines to be included, led to overall better supervision output.

Weaknesses in definition of MSRs criteria. The evaluation found that the standards applied for fiduciary issues are not defined in sufficient detail in the MSRs. The issue of irregularity in project auditing is a case in point. The definitions of implementation support and human dimension in the MSRs are equally vague relative to IFAD's long and extensive experience in rural poverty reduction.

Supervision quality under MSRs criteria. Overall, the DR found the MSRs have been met with some variations among its components. For core supervision functions, CI performance was overall good with little variance among the CIs. For implementation support greater differences were found, with World Bank, IFAD direct supervision and UNOPS showing the best performance. All CIs and IFAD direct supervision found the human dimension more difficult to handle though IFAD and UNOPS have a certain edge. To a large extent this is due to the lack of clarity of the concept and various interpretations by various CIs. CI performance with respect to supervision administration was rather low. This covers supervision planning and strategy for the whole project. In sum, the DR concluded that the MSRs introduced in 1997 were generally met though at a lower level than had been anticipated given the expectations of the GC decision. The evaluation also found that the supervision process, its expected standards and emphasis were not sufficiently documented or described in the Appraisal Reports (ARs), the loan documents and the letters of appointment.

IFAD specific aspects and strategic imperatives. Supervision fared lower in addressing IFAD specific aspects. UNOPS and IFAD direct supervision performed relatively better, due to their closeness to the evolving IFAD priorities (UNOPS not being an IFI with a specific mandate). On the strategic imperatives, no clear pattern could be discerned. The generally poor ratings seem largely the result of unclear definition and a lack of institutionalised and systematic communication on these imperatives by IFAD to the CIs and their staff (neither the CIs nor IFAD were held accountable on this particular aspect by the evaluation analysis). The World Bank scored better in linking supervision with policy dialogue whereas UNOPS was best in emphasising impact achievement. This analysis also indicated that increasing the size of SMs, especially by including a sociologist and/or gender specialist, improved the prospect for better addressing the IFAD specific aspects. The time SMs spent in the field was a positive factor in addressing the specific aspects.

IFAD-specific aspects were understood differently among IFAD regions. For example participatory approaches were better appreciated and implemented in Asia and Latin America. Similarly, gender issues were not only understood differently in the different regions but mechanisms of support and guidance provided for them varied by CIs in the various regions

Relation between supervision quality and project performance. Quantitative analysis revealed an apparent weak influence of supervision (core as well as implementation support) on project performance. Regression of average project performance rating for each CI in each region (as per CPMs assessment) against the assessed supervision rating showed only a slight positive correlation between supervision ‘quality' and project performance.

The Focus of ARs and the loan agreements in outlining IFAD specific aspects and its implementation mechanisms was found to be an important factor in the attention eventually attached to these aspects during supervision. However, detailed description in ARs is not sufficient to ensure that SMs will highlight these issues, as this depends on the quality of the SM team.

The cooperation agreements and letters of appointment for the CIs are vaguely worded, do not refer clearly to the MSRs nor IFAD's specific requirements in supervision, hence do not provide incentives for good supervision. This is particularly significant for UNOPS and the World Bank in supervising IFAD-initiated projects. Many of these agreements have not been updated since the signature of the original version.

Perspectives from the Field

Overall Views from the Field. Project managers' views on the supervision process indicate that, overall, the process is working quite well and is highly appreciated with high ratings given to reporting, meeting core supervision requirements and assisting to achieve project goals. Project managers feel that supervision is a positive experience and helps them in project implementation. Project management assessment of supervision was systematically above average and higher than the DR estimate. They scored IFAD direct supervision best in terms of achieving the MSRs, as well as the IFAD-specific aspects. About three quarters of project management expectations from supervision are in the area of implementation support, not the core/mandatory aspect of supervision. The most frequent expectations relate to advice and guidance on achieving and assessing impact, targeting women and participatory monitoring.

Concerns of project Managers. There remained concerns by project managers in specific areas of supervision: they expressed the need for:

• more participatory supervision relating to more involvement in the preparation of SM TORs and the selection of SM specialists and more intensive interaction with the target group and project staff during SM;
• inclusion of more appropriate skills mix on the missions particularly to cover IFAD's specific aspects;
• better, more frequent access to implementation support sources at local level;
• more reliance on in-country or regional resources in providing implementation support and invest in their capacity enhancement;
• greater support in developing their Monitoring and Evaluation (M&E) systems particularly for participatory monitoring and impact assessment;
• flexibility in supervision frequency as per project needs with two supervision missions per year particularly in early project implementation and for projects covering a large geographic area.

Preference for IFAD direct supervision. While project managers assessment for the main service providers was not significantly different and was above average for all five CIs, IFAD direct supervision appears most appreciated by IFAD supported projects because of:

• the direct involvement of the CPMs and thus the elimination of the ambiguity in the respective roles of IFAD and the CI;
• more effective consultations with government and better influence on decision makers;
• a generally faster response to project queries;
• the larger teams and more frequent supervision missions used under IFAD direct supervision which appear to have given better services to the projects concerned.

Achieving MSRs and IFAD specific aspects. Project manager response indicate that MSRs are overall achieved by all main CIs. The perception of project managers with regard to the supervision of IFAD specific aspects is that SMs on the whole address these aspects satisfactorily, but participation by the target group and gender sensitive implementation are the least satisfactory. Evaluation field visits, in particular, indicate that the expectations of project management regarding IFAD-specific areas in general are driven by a realism with which they have to address these subject matters every day and a realization of the extent of difficulties involved.

The field visits also suggested that the role and responsibilities of the field partners (governments and implementing agencies) in project supervision should be outlined more specifically in the ARs and other loan documentation. In particular project managers felt that SMs could be used for helping to resolve pending problems with higher level officials. IFAD's advantage in this area was the clearest.

IFAD Staff views

CPM assessment of CI performance. Within PMD's internal project portfolio review, IFAD's CPMs are requested to rate the CI's supervision performance for each project. No rating is provided for direct supervision. The CPMs rated the performance of the two CIs with the largest portfolios, UNOPS and the World Bank, as well as BOAD, most favourably followed by AFESD and CAF. In the view of the CPMs there was, on average, a reasonable supervision performance for the main CIs though many areas were pointed out for improvements. For the weaker ones IFAD stepped in to overcome identified weaknesses through the use of IFAD implementation support instrument.

Relations with CIs. All CPMs view supervision as a process to provide feedback on whether a project is in line with its objectives, to recommend corrective actions and to ensure adherence to the fiduciary aspects. But significant ambiguities remain among CPMs about the extent of their involvement in CI supervised projects. This highlights the "grey" areas, which exist in the respective responsibilities of the CIs and the CPMs. A large majority of IFAD staff believes that: (i) CI responsibilities should be more clearly defined in legal documents; (ii) a more specific and detailed letter of appointment tailored to each project is needed; (iii) clear performance indicators for CIs should be developed and the CIs held accountable to them; (iv) IFAD should develop and provide detailed guidelines for supervision of IFAD specific areas of concern and should enhance the capacity of CIs through training in these areas; and (v) the CIs need guidance in supporting the project to establish a better system to monitor and assess project impact.

Minimum supervision requirements. CPMs recognized that CIs on the whole meet the MSRs. However, a high majority of staff (75%) do not find the MSRs particularly useful and attest to their inadequacy regarding IFAD specific aspects. Many CPMs were not completely clear over the exact content of the MSRs.

Current supervision modalities. These were not found satisfactory for IFAD staff. A good majority (71%) believes that IFAD should not continue with the modality prevailing currently nor with the IFIs type supervision being used. Many argued that IFAD needs to develop different modalities based on building local capacities for implementation support and a form of IFAD's field presence. They stressed that supervision should be undertaken in close partnership with all stakeholders and use locally based resources to reflect the changing nature of IFAD supported projects (see also paragraph 30). CPMs believe that IFAD has become too dependant on one cooperating institution (UNOPS), and that the Fund should promote more diversification and competition amongst the CIs to enhance the quality of service. A large majority (71%) would like to see the projects supervised by UNOPS decrease. Some argued that as most CIs have no presence on the ground nor do they have real ownership for IFAD initiated projects, they cannot provide the desired supervision quality. Several experienced staff members observed that two (full) SMs per year gave a much better result, particularly during the first 2-3 years of a project.

Supervision quality. A strong interest (82% of staff interviewed) was expressed in strengthening PMD's internal portfolio review mechanism to improve supervision quality. ARs and Project Implementation Manuals (PIMs) need to address issues essential for good supervision more explicitly to help CIs carry out their task, facilitate monitoring and promote accountability. CPMs believe IFAD specific aspects are generally not well covered in supervision because these issues are not properly covered in the legal agreements between IFAD and the borrowing governments, and between IFAD and the CIs.

View of IFAD Staff regarding future supervision modalities. Two thirds (65%) of IFAD staff believe that direct supervision is the preferred option. The same figure for the group of CPMs that currently directly supervise the 15 pilot projects was as high as 80%. About one third of the total staff interviewed rated direct supervision as their least preferred option. This reflects different views as to what are, and should be, the CPM's priority work areas and their involvement in the supervision work of the CIs. CPMs not involved in direct supervision were generally less enthusiastic about direct supervision than those who are currently doing it, suggesting that the experience with direct supervision for the responsible CPM has had significant rewards. The CPMs undertaking direct supervision were very conscious of the additional workload arising from direct supervision as well as the difficulties encountered in mobilising the required resources, yet most of them are professionally satisfied and stimulated by direct supervision and believed in its worth for achieving better quality supervision and eventually better results on the ground.

IFAD's culture and the role of supervision. Many CPMs emphasized that IFAD still needs to shift its culture from design to implementation and impact achievement. To achieve this shift resources allocated to implementation phase should increase and changes should also be effected in the supervision modalities which would reflect the new trends in the way IFAD undertakes its business. Reference was made to the shift in IFAD projects from input/output orientation to process orientation and community-driven approaches, the emphasis on building local level institutional capacities, undertaking policy dialogue, enhancing local level partnership and empowering the rural poor. The quality of implementation support, embodied in supervision, should be consistent with this new orientation. IFAD should be more open to other more innovative options for supervision, and link it with the current IFAD efforts to develop its field presence.

Cost of supervision

The estimated cost of supervision is based on the work of IFAD's Office of Internal Audit for the two years 2001 and 2002. Cost of supervision of IFAD supported projects include: (i) fees paid directly to CIs; (ii) the IFAD staff costs incurred to oversee support and implement supervision by CIs and by IFAD; and (iii) the cost of implementation support to the supervision process through IFAD follow up budget and grant funds. The cost of IFAD staff time, in terms of salary, pension etc., has been imputed using estimated time inputs from CPMs and other staff. For 2001 and 2002 the average supervision cost ranged from USD 22 874 to USD 89 873 for directly supervised projects with an average of about USD 60 000 for CIs and IFAD. The average cost of the World Bank and of UNOPS for IFAD initiated projects was 74 254 and 68 682 USD respectively.

The analysis of supervision effectiveness, and supervision costs indicates that, on the whole the best performing supervisors (CIs or IFAD) are those with the higher cost. This is an approximate indicator that IFAD is getting proportional benefits from its resource allocation to supervision. There is, however, still quite a scope for enhancing supervision quality.

UNOPS, compared to the World Bank in particular, provides a satisfactory standard of full supervision services considering the lower level of fees that IFAD makes available for this CI. The extra costs of World Bank supervision on IFAD initiated projects are only partially offset by improved mandatory supervision services. Some of the regional co-financing CIs including AFESD (with below average supervision fees) perform well on the core supervision functions but need additional implementation support by IFAD to achieve an overall acceptable standard. CPMs have been active and imaginative in obtaining funds from grant resources and supplementary trust funds to provide additional implementation support to these institutions (see paragraph 47).

Cooperating institutions perspectives

Both UNOPS and World Bank staff highlighted the importance of informal relationships between CPMs and the supervision staff relative to (the generally irregular) formal interaction between the organizations. Both institutions suggested a desire for greater clarity in spelling out the respective responsibilities. In particular, UNOPS stressed the need to clarify IFAD expectations and welcomed the provision of more detailed guidelines and training on IFAD specific requirements. UNOPS maintains that its fees are set at a relatively low level in relation to IFAD's requests. This and the high work loads of its staff and their frequent field travel led some UNOPS staff to view IFAD as a "high maintenance client". The majority of UNOPS staff expressed the view that the relationship is increasingly dealt with as "client service provider" in the strictest sense not as a partnership for poverty alleviation.

UNOPS Portfolio Management Officers (PMOs) saw supervision as ‘facilitating an evolving process' to deliver successful projects⁴. Implementation support was seen as an important task but views varied on how well this could be achieved by UNOPS within the limited time and resources available for SMs. It was observed that supervision should identify where implementation support is needed and IFAD should mobilize resources on an ongoing basis to provide the inputs required. This is particularly relevant for building institutional capacity as it is a long term process. Supervision processes⁵ in UNOPS are not fully documented (though guidelines on loan administration exist) and new staff rely on more experienced staff for support. In offices with a low rate of staff turnover this is not a major problem, but other offices have, and are still, experiencing high staff turnover⁶. This places more pressure on the more experienced UNOPS/PMOs. A higher turnover of staff, resulting partly from the high work load and constant travelling, as well as lack of appropriate incentive framework, have reduced the quality of supervision. The major concern expressed, aside from the limited cost allowance, is that IFAD does not have a systematic process to communicate its priorities on IFAD specific aspects and IFAD strategic imperatives to the CIs.

The World Bank uses a similar but more comprehensive supervision process than UNOPS. The supervision guidelines are extensive and are supported by specific financial management, procurement and safe guard procedures and documentation. The World Bank has clear accountability and quality assurance management processes in place for supervision. World Bank staff are very clear that supervision processes on IFAD projects have to be consistent with World Bank projects. Supervision inputs must cover three main areas: (i) fiduciary issues; (ii) ten safeguard requirements; and, (iii) implementation support. The first two areas were the highest priority given available resources.

As IFAD developed its capacity and expertise in certain areas, pressure has been applied by some CPMs on the World Bank to address IFAD specific aspects as part of supervision implementation support. Often the relatively small IFAD funded component(s) is more complex, geographically remote and with emphasis on community based activities and local institutions building. This requires additional time and skills inputs that the World Bank argues is not paid directly for. The World Bank staff see advantages in working with IFAD on projects targeted at rural poor, but they argue that IFAD will need to provide additional resources to ensure IFAD's specific interests are covered in projects that are initiated and supervised by World Bank. World Bank staff also flagged an apparent lack of communications and feedback processes, including systematic apprising about the changing IFAD priorities. This aspect assumes greater importance with the IFI CIs if IFAD is to succeed in influencing these institutions toward the goals of rural poverty reduction.

UNOPS has and can adjust its approaches to meet IFAD requirements but the World Bank staff are somewhat more constrained by the requirements to meet World Bank mandatory supervision requirements and safeguards. Although UNOPS has staff dedicated to IFAD work these staff are not included in the other parts of the programme cycle, e.g. design efforts, IFAD's internal portfolio review, and IFAD fora concerned with developing future vision and strategy.

Implementation of the five-year Plan of Action

Implementation of the Action Plan has been substantial but not complete as summarized below:

1. MSRs have not been incorporated into formal and legal documents with the CIs. For core supervision functions, MSRs are fulfilled, to varying degrees, by most CIs. However, IFAD specific concerns are not being adequately addressed by the MSRs and some are dealt with in a vague manner. CIs performance relating to these concerns is lower. There is no clear system to hold CIs accountable for MSRs.

2. Coordination Procedures between IFAD and CIs have not been systematically addressed with informal cooperation between CPMs and their CI counterparts still the basis for coordination and effectiveness.

3. Portfolio Management. Good progress has been made and a more efficient portfolio management system is in place. PSRs are prepared as part of the yearly regional portfolio reviews with a consolidated Project Portfolio Review (PPR) presented annually to the April Executive Board (EB). The "realism" of estimates of project and CI performance could be improved through an agreement on clearer more detailed indicators to facilitate monitoring. A separate accounting system to track the actual cost of direct supervision has been developed, but is not used systematically and effectively.

4. Strengthening Learning Loop. The main learning linkage continues to be an informal and loose process. Consultants working directly with CPMs undertake implementation follow up and project design/review work. No formal mechanism was put in place to capture learning from CI and IFAD supervision experiences. Given staff workload it is not evident how effective is the informal process of exchanging knowledge.

5. The Direct Supervision Pilot Programme has been fully implemented using modalities ranging from the UNOPS model to large consultant teams mobilized twice per year. Average annual direct supervision costs are higher than those for other supervision modalities. Systems to monitor and assess improvement in project performance through direct supervision have not been put in place nor have mechanisms to ensure that the value added by direct supervision experience is shared and disseminated in-house and with CIs.

Main conclusions

Supervision in IFAD's Project Cycle. Institutional attention to supervision is a reflection of IFAD's strategic emphasis and priorities with respect to various stages of the project cycle. The dominance of design (and pre implementation) cost in IFAD's project cycle is a manifestation of an underlying institutional priorities to project approval stage. IFAD efforts to shift its emphasis to implementation stage and impact achievement have to be enhanced. One way of doing this is through strengthening the supervision processes.

Supervision Effectiveness. A large cross regions and cross CIs assessment by the evaluation team attested to a reasonable level of supervision overall though with variations between the various tasks making up project supervision and between the CIs. Consistently CIs have performed better on fiduciary aspects whereas implementation support, particularly for IFAD's specific requirements (and strategic imperatives), lagged behind. A similar view as to the effectiveness of supervision was found among IFAD operational staff. They also expressed strong views regarding the need to modify existing supervision modalities.

The supervised clients (project managers), on the other hand, expressed remarkable satisfaction with supervision of their projects. They ranked the services they receive from supervision at a consistently higher level (between fully and highly satisfactory), but services received from IFAD's direct supervision ranked best. However, they strongly expressed the need for more frequent and better access to local level implementation support, more participatory supervision and for changing supervision frequency with the nature of the project and the implementation stage.

On the whole, UNOPS, the recently started IFAD's direct supervision and the World Bank showed a stronger supervision performance than the regional, smaller CIs. IFAD direct supervision followed by UNOPS ranked systematically better in implementation support. CAF and AFESD were found relatively less effective than the other CIs. Smaller regional IFI/CIs notably CAF and BOAD are receiving closer attention by the regional divisions in terms of building their capacity and there are recent emerging signs that these efforts may pay off. All perspectives indicated that while a number of areas require attention from IFAD to improve supervision, the IFI model in supervision, currently prevailing in IFAD, is not likely to deliver the quality of supervision needed under IFAD changing priorities and nature of field operations. An innovative break through is needed to move supervision to a higher plateau of performance.

Implementation of the five year Action Plan. While attention to supervision in IFAD has no doubt improved after 1997, the changes called for by the Five-Year Plan of Action have only partly been achieved. Progress was concentrated in the PPR/PSR processes and, most important, in the introduction of direct supervision. The MSRs were developed, and are currently met, but the standards they embody are below what an institution of IFAD's mandate and experience in poverty reduction should be expecting. There remains an important agenda to be pursued. MSRs – or any equivalent form of common guidance for supervision of IFAD supported projects – must be made the working instrument that ensures consistency between IFAD specific interests and what is pursued on the ground. Some other unfinished agenda also remains from the 1997 Plan of Action in the areas of coordination between IFAD and the CIs and reinforcing the learning loop between design and implementation.

Project portfolio management. Very good progress has been made in portfolio management. This has enabled IFAD's management and its EB to focus attention on critical portfolio concerns and to take necessary actions. It also has helped to bring an institutional perspective to what otherwise CPMs might have looked at from single project and/or country angles. Nevertheless there remains some short comings about the PPR process as a vehicle for improving the quality of supervision. Renewed and more intensive efforts are needed in these areas to enhance supervision quality and overall accountability in delivering supervision services. IFAD's Controller's Office remains concerned about the handling of audits.

Concentration of supervision and the IFIs model. Concentration of project supervision into fewer CIs has continued in recent years. UNOPS is currently responsible for supervising about 60% of ongoing projects and is the only non IFI CI. But the IFIs model of supervision⁷, also used by UNOPS, has gone largely unchanged. Because of the changing way of conceptualizing and designing projects in IFAD, the IFI model in supervision may not be the most effective modality to enhance project performance. The evaluation found some interesting pioneering work (notably in PL) done through the use of regional technical assistance grant (TAG) funded programmes for capacity building of local and regional institutions to provide significant inputs into the supervision process in support of project implementation. Some CPMs are also pioneering local level partnerships a proxy for field presence to promote the right type of project implementation support. There is an opportunity to extend and develop further innovative supervision modalities that are especially suited to the needs of IFAD assisted projects in different regions.

The role of implementation support by IFAD. Implementation support, to supplement CI supervision, has existed in IFAD from the very beginning and is firmly established as an input in the supervision process. Despite its importance and the large amount of resources it absorbs (more than one quarter of supervision costs), IFAD has not developed a clear operational policy and priorities on which to anchor this concept. The evaluation did not find any direct correlation between implementation support and CI performance nor with project performance. Most importantly the use of implementation support is not linked with a clear policy of local and national capacity building. This may not be consistent with sustainability requirements.

IFAD supported projects expressed the desire for more frequent interaction with locally-based implementation advisers, particularly for process oriented community driven projects (currently the majority of IFAD projects). There is a need for a re-examination of the concept of implementation support in IFAD (and for the CIs), re-position it within a medium-term strategy for strengthening local capacity towards the support of community-driven projects and adopt a more systematic approach in its execution.

Cost of supervision. The results of cost analysis for all CIs over two years (2001 and 2002) demonstrated that the best performing CIs are those with the higher supervision cost. As various CPMs observed "IFAD gets from CIs what it pays for". While this is an approximate indication that benefits are in line with cost incurred, no conclusive statements can yet be made (particularly regarding IFAD direct supervision) as this would require, inter alia, analysis over a longer time horizon. Difficulties in data availability and compilation prevented the undertaking of such exercise.

Direct supervision. The evaluation preliminary findings noted many positive features in the performance of direct supervision. This was confirmed by the DR and the project managers survey. The evaluation also noted some tradeoffs, in particular in terms of the high unit costs of direct supervision. However, the evaluation could not draw any firm conclusions regarding direct supervision as the average implementation period of the directly supervised projects and the disbursement rates were much below other projects reviewed.

Recommendations

The evaluation recommends that actions are taken to shift the boundaries of the supervision modalities for IFAD towards a new model that better reflects the nature and needs of IFAD supported projects. It is recommended that actions are taken at two levels simultaneously: (i) Policy Level and (ii) Operational Level.

Policy level

Recommendation 1: develop a new oolicy and regional programmes for supervision in IFAD. The need to introduce improved supervision modalities reflecting the changing priorities and nature of the Fund's supported projects has become evident through the evaluation.

A number of pioneering initiatives are now under way in IFAD at regions and/or CPMs' level. The evaluation recommends that IFAD builds on these initiatives and develop a new medium term IFAD policy for supervision. The policy should combine two crucial dimensions of IFAD's emerging priorities: building local/regional level capacities and promoting innovative partnership with stakeholders at local level for pro-poor implementation support. The policy should specify the principles involved in designing and implementing regional programmes that extend and develop improved supervision modalities taking into consideration the variance in institutional capacities in different regions, and linking explicitly with the IFAD ongoing efforts regarding its field presence. An implementation plan with measurable indicators would have to be associated with the development of such a policy, including at the level of the different regional divisions programmes. While the results of the policy are gradually materializing, IFAD should continue to address areas needing strengthening under the current modalities.

Operational level

Recommendation 2: tevise and update IFAD's minimum dupervision requirements for its CIs and hold CIs firmly accountable to the MSRs

The number one recommendation of the 1997 Five-Year Plan of Action, is in need of renewal and reinforcement with respect to (i) its transformation into a dynamic system that reassures IFAD that emerging new concerns and priorities are reflected in supervision standards; (ii) core supervision and fiduciary aspects and IFAD specific requirements to be spelled out clearly and their standards specified; (iii) cooperation agreements and letters of appointments with CIs to rigorously reflect the new MSRs (or new standards) with special emphasis on IFAD specific aspects and strategic imperatives; (iv) adherence of CIs to these principles to be closely monitored (see also recommendation 5).

Recommendation 3: build CI capacity in the supervision of IFAD's specific aspects and provide appropriate tools

It is IFAD's responsibility to strengthen the capacity of the supervisors of its supported projects (CIs and CPMs). Written guiding frameworks, periodic joint CI/IFAD training sessions and sharing of learning and best practices between CIs and between IFAD and CIs can lead to great improvement. Training of project staff themselves in these aspects is an important dimension of capacity building and should be considered a priority area for implementation support by IFAD.

Recommendation 4: assess and revise IFAD's implementation support patterns and practices

It is critical to realign the use of all IFAD instruments used in supervision towards capacity building and local level partnership for better project implementation. It is recommended that PMD undertakes an assessment of implementation support practices in IFAD and their effect on project performance and to establish on these basis clear principles and criteria for the use of this instrument and monitor it closely to ensure that resources are allocated in areas of maximum returns for project performance.

Recommendation 5: Improve supervision quality assurance in IFAD

It is recommended to strengthen the PRR process by adding a strong institutional focus on improving the quality of supervision and facilitating monitoring of CI performance. This would increase the confidence in the reliability of the PPR system as a portfolio monitoring mechanism and enable IFAD to give a better informed feedback to CIs. Under such a focus IFAD would introduce a vehicle to provide incentives for good supervision, enhance supervision accountability for meeting well defined standards and improve results on the ground. This should apply to CI as well IFAD direct supervision. The latter is particularly important as no system exists at the moment within PSR to assess the performance of supervision in directly supervised projects. CIs would have to be incorporated in such quality assurance process to ensure ownership of recommendations. Reviewing other IFIs practice in this respect would help identify good practices.

Recommendation 6 strengthen the learning loop from supervision

Much more needs to be done for a systematic strengthening of the learning loop between design and implementation across and within projects. A more systematized learning process has to be developed, involving the principal actors involved with supervision on the one hand and with project formulation/appraisal and the project portfolio review mechanism on the other. It seems especially appropriate to draw on UNOPS PMOs who have specialized experience with the implementation of IFAD supported project as well as supervision consultants.

Recommendation 7 improve coordination between IFAD and CIs and strengthen the partnership dimension

This continues to be a matter of concern for IFAD and the CIs, and to some extent project managers. Institutionally the agenda for improvements in this domain should eliminate areas of overlap, clarify responsibilities, improve communication, establish a more consistent reporting system across all CIs, and expand the view about CIs from mere service providers to partners for rural poverty reduction with joint responsibility for outcome and stronger ownership. Appropriate incentives for supervisors are of essence to develop such ownership.

Recommendation 8 exercise realism in setting up fees for the CIs and base it on project needs

If the scope of supervision is to be consistently maintained or even enhanced, including the practice of an average of two supervision missions at least at the initial phase of project implementation, of a longer stay in the project area, and adequate interaction with beneficiaries, realism in the setting of fees is essential. Implementation support, a subject recommended for review under recommendation 4, is a complementary resource in strengthening supervision services performed by CIs and would have to be considered in the context of setting realistic CI fees.

---

1/ The Corporate Level Evaluation Team comprised: Ms Mona Bishay, Deputy Director Office of Evaluation (Team Leader); Mr Ian Teese (Senior Evaluation Consultant/Agricultural Economist); Ms Maliha Hussein (Evaluation Consultant/Rural Sociologist); Ms Sandra Romboli (IFAD Consultant); and Mr Hans Wyss (Special Resource Person). The team was also supported by three consultant reviewers: Jane Blaxland, Hamdy Eisa and Octavio Damiani for the French, Arabic and Spanish reports respectively.

2/ The Agreement Establishing IFAD states that the Fund shall delegate this role to cooperating institutions: "The Fund shall entrust the administration of loans, for the purposes of the disbursement of the proceeds of the loan and the supervision of the implementation of the project or programme concerned, to competent international institutions". (Article 7, Section 2(g) of the Agreement Establishing IFAD).

3/ 1994 version, paragraph 43.

4/ The role of the CIs were modified in the new General Conditions produced in 1998 to add the facilitation role of supervision. This had apparently not been clearly communicated to some PMOs.

5/ In addition to loan and procurement administration, new PMOs expressed a need for further training in high level negotiation and change management skills.

6/ By mid 2003, UNOPS Kuala Lumpur office will possibly have three out of four IFAD PMOs with less than 12 months supervision experience.

7/ With supervision done through external missions once or twice a year concentrating mainly on fiduciary aspects.