Introduction

Background. In February 1997, the IFAD Governing Council adopted resolution 102/XX on Loan Administration and Supervision of Project Implementation, together with a Five-Year Plan of Action. The resolution stated that IFAD "may supervise specific projects and programmes financed by it". According to the action plan, no more than 15 IFAD-initiated projects were to be directly supervised and administered during the five-year period. This initiative, including the 15 projects, has since been referred to as the Direct Supervision Pilot Programme (DSPP). The Governing Council resolution entered into effect on 21 February 1997 and will cease to be operational five years after the date of effectiveness of the last approved project to be directly supervised by IFAD¹. The Governing Council also decided that the DSPP should not entail any cost increases and that IFAD should use the same funds that would have been spent on supervision by cooperating institutions (CIs) to cover its direct supervision activities.

In order to assess the results of the action plan, in 2002/2003 the Office of Evaluation (OE) undertook an Evaluation of the Supervision Modalities in IFAD-Supported Projects (ESM). The objectives of the ESM were to "evaluate the effectiveness of current supervision modalities against the minimum supervision requirements [MSRs] … and other indicators of quality, and review the achievements under the Five-Year Plan of Action". It is important to note that most of the directly supervised projects were in their early stages of implementation at the time of the ESM. Hence, although the ESM provided concrete findings on supervision through CI, it provided only an overview of the emerging characteristics of IFAD's direct supervision efforts.

DSPP objectives. The overarching objective for embarking on the DSPP was to enable the Fund to acquire first-hand knowledge from supervision activities and to incorporate lessons learned from ongoing operations more effectively into its project design work. It was also to provide IFAD with "knowledge of the supervision function, of what are the costs of an adequate project supervision and of the development impact and human dimension of the projects in its portfolio. The Fund's involvement in direct supervision would also complement and improve cooperating institutions' own activities (mainly the human dimension of projects/programmes)". Although enhancing development effectiveness² was not an explicit objective, direct supervision was expected to contribute to improving implementation performance and project impact.

Evaluation objectives. The main objective of the corporate-level evaluation (CLE) was to make an overall assessment of the DSPP's achievement in enhancing the implementation and impact of IFAD-funded operations. More specifically, the evaluation had the following key objectives: (a) compare and contrast direct supervision by IFAD with selected, relevant examples of supervision undertaken by CIs; (b) examine the processes established, alternative approaches and the experiences of country programme managers (CPMs) in undertaking direct supervision. This would include comparison with the approaches, systems and experiences of other international financial institutions (IFIs); (c) assess the efficiency of the direct supervision modality; and (d) examine the systems established to capture the experience and insights from direct supervision and the ways in which this has been of benefit to IFAD's project design processes and implementation support activities.

Evaluation methodology. The approach for the direct supervision evaluation has been to build on the methodology and results of the ESM. The criteria³ used in the CLE to assess the DSPP include: (a) the relevance of direct supervision in relation to the DSPP's objectives at the time of approval but also in today's context; (b) the effectiveness of the DSPP, measured against the achievement of the stated objectives of the programme and using the indicators specified for measurement of the impact of direct supervision;⁴ (c) the efficiency of direct supervision; and (d) an analysis and comparison of the actual and potential impact of the projects included in the direct supervision pilot programme with those supervised by CIs.

The evaluation process was planned to allow for triangulation of evidence and the views obtained from the main actors in the DSPP (the key government focal point at the national level, together with implementing agencies, beneficiaries, and the IFAD staff concerned). Moreover, OE had completed evaluations of three directly supervised projects⁵ in the past few years, which provided valuable additional sources of information and assessments. Since 2003, OE has also undertaken three country programme evaluations⁶ that included assessment of one directly supervised project in each of these countries. Relevant information and reports from the Independent External Evaluation (IEE) of IFAD, which analysed in detail two directly supervised projects⁷, have also been utilized. Finally, four projects⁸ in the evaluation control group have either been evaluated by OE or included in the IEE.

Evaluation scope. The evaluation involved a comparison of the directly supervised projects with 15 projects supervised by CIs (the latter therefore represented the evaluation's control group). That is, three CI-supervised projects per region were included in the control group based on a set of selection criteria agreed with the evaluation's core learning partnership (CLP)⁹. The list of projects included in the DSPP and the control group for the purposes of the evaluation can be seen in Appendix I.

Evaluation process. The CLE process began with the preparation of an approach paper, which provided an opportunity for developing a solid understanding of the evaluation's objectives, scope, methodology, time frames and expected outputs. The evaluation benefited from the views of the CLP. An external reviewer¹⁰ with wide experience in project supervision issues was contracted to advise OE at critical phases during the evaluation and to review key evaluation outputs. Moreover, OE undertook a thorough peer-review process within the division to improve the evaluation's overall quality.

The evaluation included the following activities: desk reviews of the 15 directly supervised projects and 15 projects in the control group; interviews with IFAD management and staff; field work in 13 of the 15 directly supervised projects and in eight control group projects¹¹; discussion with the United Nations Office for Project Services (UNOPS) and selected IFIs at their headquarters and at the country level; direct supervision cost analysis; preparation of an early feedback note on an experimental basis, with the objective of sharing the emerging evaluation results and sensitizing IFAD management before the draft evaluation report was ready for discussion; organization of a round-table workshop to discuss the evaluation's results and lay the foundations for the agreement at completion point of the evaluation; and discussion at the September 2005 sessions of the IFAD Evaluation Committee and Executive Board.

The draft evaluation report was shared for review and comments with: staff in all the projects included in the DSPP and the control group, IFAD management and other staff, selected IFIs and CIs, and the government officials concerned at the national level in countries in which IFAD has funded a directly supervised project. Their comments have been included in accordance with the provisions in paragraph 42 of the IFAD Evaluation Policy.

The evolution of direct supervision of IFAD

Origin of direct supervision. The Executive Board reviewed the Joint Review on Supervision Issues for IFAD-Financed Projects in December 1996. The report was submitted to the Governing Council in February 1997, together with the Five-Year Plan of Action for the DSPP. The plan¹² included the following actions, together with time frames for their implementation by IFAD:

• "agreement by Governing Council for IFAD to supervise test projects;
• criteria to be used for selection of test projects to be submitted to Executive Board;
• 15 test projects to be determined (3 projects per region);
• identify and negotiate with reputable private national or international organizations to undertake procurement and financial administration;
• progress report of test projects to be reported to Executive Board;
• analytical accounting system to be established to track the actual cost of direct supervision as well as supervision by CIs;
• mid-term review of supervision of test projects to be submitted to Executive Board; and
• establish monitoring system to evaluate the test projects."

DSPP modality. One important feature of the DSPP has been the cooperation between IFAD and UNOPS. The latter was contracted in July 1998 by IFAD to undertake the supervision of fiduciary aspects (such as procurement of good and services, disbursements, ensuring compliance with auditing and financial requirements) in the context of the DSPP. UNOPS was paid a standard amount equal to USD 12 000 per year per project for their services in the specified areas. IFAD's specific role in the DSPP was thus to arrange and conduct supervision missions, organize the necessary follow-up and provide the overall implementation support required by borrowers and their projects.

Monitoring and progress reporting. IFAD management presented indicators for the measurement of the impact of direct supervision to the Executive Board in April 1999 (see Chapter II, Box 1). IFAD further committed to reviewing the indicators at least one year after most directly supervised projects had become fully operational. For comparative analysis, the Report of the Joint Review on Supervision Issues for IFAD-Financed Projects explicitly requested that a control group of CI-supervised projects, similar in other respects to the directly supervised projects, be identified and monitored.

In terms of reporting, the Governing Council decided that the progress, lessons learned and results of the pilot programme would be reported annually to the Executive Board. Moreover, IFAD would conduct a mid-term review (MTR) of the supervision of test projects, also to be submitted to the Board. In approving OE's work programme for 2004 during its session in December 2003, the Executive Board requested OE to undertake a CLE on the DSPP. Finally, the Governing Council decided that the President shall submit the results of IFAD's experience and conclusions on the DSPP to the Executive Board for its review before the corresponding resolution would cease to be operational (i.e. in June 2006).

Implications of selected new IFAD initiatives. The growing emphasis on ‘country presence'¹³ will have important consequences on the modus operandi and costs of both direct supervision and supervision by CIs. For instance, IFAD's country presence will have a role in supervision activities, ranging from simply facilitating the organization of supervision missions to the more substantive role of providing backstopping and follow-up on implementation issues. Furthermore, management is developing a proposal for the Fund's new operating model. There are two elements in this model that are likely to have important implications for IFAD's supervision activities. These are: (a) the shift in the unit of account from the project to the country programme level; and (b) the utilization of the Results and Impact Management System (RIMS) to capture and analyse data systematically according to a core set of indicators. IFAD will need to pay attention to these aspects in its future supervision activities.

Direct supervision analysis

Selection of the pilot projects. Most of the 15 projects were selected according to the criteria presented by IFAD to the Executive Board in September 1997. Two projects (in Gaza and the West Bank and Zimbabwe) have been adversely affected by political instability. While the inclusion of the Zimbabwe project in the DSPP was justified, because the unexpected events in the country could not have been foreseen at the time of Board approval, the inclusion of the Gaza/West Bank programme in an unstable country situation did not reflect an appropriate choice for inclusion in the pilot programme. The evaluation also notes that the Bangladesh project was not an appropriate choice in that it was not innovative, replicating broadly the design of a previous IFAD-financed project in the country.

Approaches to direct supervision. There have been various approaches to direct supervision in the 15 pilot projects. These include: (a) intensive CPM involvement in all steps of direct supervision, including fiduciary aspects. In these cases, the role of UNOPS was largely limited to disbursement processing; (b) the CPM completely delegates fiduciary aspects to UNOPS and focuses on implementation support matters, although s/he may not write mission reports and may not be as closely involved in the annual work programme and budget processes; (c) fiduciary aspects are dealt with by UNOPS, and the CPM appoints consultants who would be largely responsible for leading supervision missions and producing the corresponding documentation. The CPM may participate in key stages of the missions. In this approach, the CPM takes a management role and focuses on mission deployment and ensuring that all outputs are produced as required; and (d) this approach is specific to those countries in which IFAD has some form of local representation that plays a part in direct supervision by participating in supervision missions and following up between missions. The distribution of the pilot projects across the four approaches has been fairly evenly spread, with three to four projects each following approaches (a)-(c) and two projects using approach (d).

Implementation of direct supervision. The evaluation reviewed the implementation experience of direct supervision activities as compared with those under supervision by CIs. Its report observes that supervision planning during preparation of country strategic opportunities papers (COSOPs) and the project design phase was given limited attention in both forms of supervision. On another issue, the evaluation notes that the continuity of CPMs in direct supervision activities is important, and that in some pilot projects, there have been more than three CPMs allocated to the project in the last five years. In terms of supervision mission frequency and duration, the directly supervised projects have received close to two supervision missions per year, against one supervision mission for the control group projects. The average mission days per project per year were 15.2 days for direct supervision as compared with 11.2 days for the control group.

The composition of supervision mission teams in the directly supervised projects and in the control group has not been significantly different, with both groups making use of local consultants. There has also been more consultation between IFAD, the country and the project authorities before deciding what expertise to include in each supervision mission, especially in the context of the DSPP. Direct supervision missions on average comprised 2.6 personnel, compared with 2.5 for the control group.

So far, eight of the projects directly supervised have benefited from MTRs, whereas 11 projects in the control group have had an MTR. This is understandable, given that the average implementation period of the directly supervised projects (4.8 years) is less than that of the control group (6.5). In the case of the DSPP, the MTR has offered an opportunity to formalize a number of decisions made during the regular supervision missions that preceded the MTR. That is, given the more constant and intensive interaction between IFAD and the project authorities in directly supervised projects, CPMs have been able to stay on top of implementation issues and make key decisions within the framework of supervision missions themselves, rather than wait until the MTR stage. Hence, the MTR does not appear to have been critical in the context of the DSPP. It has been a critical instrument in CI-supervised projects, as it has provided CPMs the possibility of conducting a comprehensive review of what has been done and of introducing the required corrective measures to project design and implementation arrangements.

The quality of supervision reports varied across the directly supervised projects. In the terms of reference of and reports on direct supervision, attention largely focused on implementation issues and on physical and financial achievements, and less on lessons learned and impact. The same may be said of supervision reports prepared by CIs. However, supervision reports of the DSPP projects provided better coverage of issues of concern to IFAD, such as gender mainstreaming, monitoring and evaluation, beneficiary participation, building of grass-roots institutions and so on. Moreover, the documentation of the directly supervised projects was more complete and more easily accessible at IFAD than that of the control group.

In terms of feedback, the evaluation concludes that the DSPP did not systematically follow the set procedures established for feedback. Often an informal mode of communication was chosen rather than following official channels. In spite of the above, the evaluation notes that there is generally a faster response to project queries and follow-up on supervision recommendations under direct supervision than under CI supervision. The feedback in supervision undertaken by CIs is also erratic. Few CIs produce all the required documentation. The inadequate type and quality of reports and feedback bring up the critical issue of the current lack of quality assurance systems in the overall supervision activities of IFAD, which will be discussed later.

No notable difference in participatory processes was observed between IFAD direct supervision and supervision by CIs. However, there appears to be a clear preference for the increasing trend of conducting joint review missions (as organized by some IFIs) with the governments concerned, as opposed to traditional supervision missions. The term ‘supervision', itself, was described by many partners at the country level as being top-down, one that did not reflect the partnership ethos between the country concerned and IFAD.

Project status reports (PSRs) are an important instrument in IFAD's overall monitoring and reporting system. In PSRs of directly supervised projects, there has been no analysis or lessons on direct supervision processes or information on costs. In addition, PSRs do not include ratings on the IFAD impact indicators for direct supervision, which were developed at the beginning of the DSPP. For CI-supervised projects, CPMs include an assessment of the CI's performance and assign a corresponding rating. For directly supervised projects, PSRs do not include a rating of IFAD's performance in direct supervision. With regard to the process in the preparation of PSRs, it must be noted that the PSRs of CI-supervised projects are done by the CPM concerned, whereas the same CPM responsible for direct supervision prepares the PSR for the project under his/her direct supervision.

One important message emerging from the evaluation is the very wide support by IFAD partners of direct supervision. Government authorities, development organizations and project authorities were all of the unanimous opinion that there were advantages in having a direct contact with IFAD through CPMs. For example, partners conveyed the view that direct communication and interaction with IFAD contribute to better implementation and a stronger partnership, be it in terms of policy dialogue, identification of future pipeline and cofinancing opportunities or knowledge-sharing. There was one case in which a criticism was not of direct supervision, as an approach, but of the frequent staff changes at the CPM level that the project had experienced. The perceived lack of seniority and qualifications of some staff assigned by IFAD for this task were also raised for some projects.

Fiduciary aspects (under the formal responsibility of UNOPS) of directly supervised projects perform better than those of the control group. This finding is consistent with the ESM finding that UNOPS tends to perform better than other CIs. As UNOPS is the only CI involved in the DSPP, it is not unexpected that there would be a higher level of performance than for the control group, which incorporates a variety of CIs.

Project implementation performance. The evaluation first compared project implementation performance using indicators and ratings contained in the PSRs for both the directly supervised projects and the control group. Thereafter, the evaluation used the same set of indicators included in the PSRs and, based on its own independent ratings, compared the implementation performance of the directly supervised projects with those supervised by CIs.

From the evaluation's analysis, it is evident that the directly supervised projects perform better than CI-supervised projects across the PSR indicators in terms of, for example: compliance with loan covenants (directly supervised projects were rated on average 3.4 as against 3.1 for CI-supervised projects);¹⁴ performance of M&E systems (3.2 for directly supervised projects against 2.6); availability of counterpart funds (3.5 against 3.2) and so on. One explanation might be more optimistic reporting by CPMs, who, as mentioned previously, are themselves responsible for the preparation of the PSRs. However, the evaluation's independent assessment also demonstrates that, on the whole, directly supervised projects have a better implementation performance as compared with CI-supervised projects across the PSR indicators. All rating scores may be seen in Chapter III, Tables 4 and 5.

The comparison of the directly supervised projects with the 15 projects in the control group was also developed by assessing the average time from approval to effectiveness. The analysis reveals more favourable performance by directly supervised projects. That is, the average time lag is 15.36 months for directly supervised projects as compared with 17.21 months for all other projects in the same country. Moreover, in terms of disbursement performance, the average rating by the evaluation team for directly supervised projects is 2.6 as compared with 1.9 for CI-supervised projects in the control group. For example, the average cumulative disbursement rate for all directly supervised projects in the fifth year of project implementation was around 62% as compared with 43% in the control group. Chapter III, Figure 1 provides more information on disbursement performance.

Finally, the evaluation reviews one indicator that is not included in the PSRs: the time overrun factor in project implementation (number of years/months a project is extended beyond the original completion date). Based on the calculations made by the evaluation, the overall time overrun for directly supervised projects is on average 0.54 years, as compared to 1.4 years for projects in the control group. Time overrun is an important indicator, as it reflects the soundness of design, but also the ability of the supervision process to recommend timely corrections and improvements during implementation, as and when required. Time overrun is also significant because it has an administrative cost implication for the Fund: additional supervision costs and related staff time need to be allocated for the time that projects run beyond their original completion date.

The human (social) dimension in supervision and attention to key areas of IFAD's catalytic role. There is evidence based on the evaluation's analysis that through direct supervision, as compared with supervision by CIs, IFAD has paid more attention to issues such as the targeting of women (rated 3.1 for directly supervised projects, compared with 2.5 for projects supervised by CIs); targeting the poor (rated 3.0 for directly supervised projects against 2.4); beneficiary participation; gender mainstreaming; participatory monitoring and evaluation and so on. These trends are by and large consistent with the ratings included in the PSRs of the directly supervised and control group projects. Direct supervision has also provided IFAD an opportunity to focus more on issues such as innovation (rated 3.2 against 2.5 for CI-supervised projects), partnership, knowledge management and policy dialogue, which according to the IFAD strategic framework contribute to improving project performance and impact. In each country exposed to direct supervision, according to the evaluation and partners at the country level, the presence of the CPM during supervision is seen as an opportunity to advance IFAD's broader objectives, such as those listed above. Chapter III, Figures 2 and 3 provide all comparative rating scores on the aforementioned indicators for the DSPP and projects supervised by CIs.

The evaluation's rating for knowledge management of directly supervised projects (rating 2.7) is only marginally better than that of projects in the control group (2.5). This is partly explained by the fact that, while the knowledge acquired through direct supervision at the individual CPM level was high, there was no systematic effort to document, analyse or share such learning from direct supervision. Moreover, no specific activities were conducted or resources allocated to knowledge management in the directly supervised projects, and their supervision reports did not emphasize lessons learned. The same is true for the projects in the control group. However, this is considered a particular shortcoming in the case of direct supervision, as knowledge management in the broader sense was a specific objective of the DSPP (which is not the case for CI supervision).

Development effectiveness. The first thing to note is that, contrary to the requirement of the action plan adopted by the Governing Council, no specific, ongoing monitoring and reporting mechanisms to trace project impact were established, neither at the individual project level nor for the DSPP as a whole. In addition, contrary to the recommendation in the Report of the Joint Review on Supervision Issues for IFAD-Financed Projects, "for comparative analysis a control group of CI-supervised projects similar in other respects to the directly supervised projects" was not identified and monitored. Nevertheless, the evaluation made efforts to analyse the development effectiveness of the directly supervised projects and compare them with projects in the control group.

The evaluation has undertaken three specific types of comparison that, taken together, provide an overview of the development effectiveness of the directly supervised projects in relation to the CI-supervised projects. Tables 6, 7 and 8 in Chapter III contain the specific rating scores that provide the basis for the analysis.

First, based on three indicators (quality of supervision recommendations, follow-up actions and support provided to the project), the overall quality of supervision was assessed (see Chapter III, Table 6). These indicators are also used by the World Bank's Quality Assurance Group in determining the quality of the Bank's supervision, which is considered an important ingredient in achieving development effectiveness. In fact, in one of their recent evaluations, the Operations Evaluation Department of the World Bank concluded that well-supervised projects are twice as likely to succeed as are poorly supervised ones. The DSPP evaluation's assessment according to the three indicators reveals a positive trend in favour of directly supervised projects. For example, quality of supervision recommendations is rated 3.3 for DSPP, compared with 2.9 for CI-supervised projects.

Second, the evaluation compared directly supervised and control group projects using the 11 IFAD impact indicators for the DSPP.¹⁵ By and large, the directly supervised projects do better than those supervised by CIs across all indicators (see Chapter III, Figure 4). For example, "identification of new project concepts for inclusion in the pipeline" is rated 2.9 for DSPP as compared with 2.1 for CI-supervised projects. However, the control group projects perform marginally better in terms of timeliness of reporting (rated 2.9 for projects supervised by CIs, compared with 2.8 for DSPP). The reason is partly that, as subcontractors, CIs pay specific attention to project reporting. This issue has not been given the same level of attention in direct supervision, given the constant contact between IFAD and the project authorities. It can also be seen that the ratings for costs appear to favour CI supervision, as supervision by CIs costs less on average (this will be discussed in more detail later).

Third, the evaluation rated the directly supervised and CI-supervised projects using the six impact domains in the OE methodological framework for project evaluation (MFE). Moreover, the two overarching factors in the MFE (sustainability and innovation/replication) have been included in the analysis (see Chapter III, Table 7). In short, the directly supervised projects are rated better in most impact domains (e.g. food security is rated 2.9 for DSPP, compared with 2.5 for CI-supervised projects), although in two impact domains (environment and sustainability), the directly supervised projects performed less well as compared with the CI-supervised projects.

With regard to the latter issue, it should be noted that 12 of the 15 projects in the control group included a specific objective or component related to environmental matters. Consequently, their supervision missions have normally adequately reviewed project progress in this area. On the other hand, only five directly supervised projects have a specific environmental component and their supervision missions have not always included the required skills to undertake a thorough assessment of environmental issues and to provide the necessary backstopping to project staff. Furthermore, environmental matters have not received the same level of focus within IFAD at large as compared with areas such as gender mainstreaming and social capital formation.

No conclusive statement can be made on project sustainability at this stage. On the one hand, the slightly less positive ratings of the directly supervised projects as compared with the control group may be explained partly by the relatively fewer years of implementation of the directly supervised projects. That is, many of the projects in the DSPP have devoted greater emphasis, at least in the initial years, towards developing grass-roots organizations and promoting participatory processes and less towards productive activities. This may be a cause of the lower sustainability rating at this stage. Moreover, some of the directly supervised projects (in Brazil, India and the Sudan, for example) suffered initial delays in implementation due to compelling political and administrative circumstances, which could be another reason. On the other hand, the IEE concluded that during the early stages of a project there are relatively high expectations that project benefits will endure, but these expectations are modified in the later stages of implementation.

Finally, the overall composite rating¹⁶ of the six impact domains is compared with the rating available in the PSR on ‘meeting development objectives'. In both cases, directly supervised projects do better than CI-supervised ones (see Chapter III, Table 8). The MFE composite index is rated 2.9 for the DSPP as compared with 2.5 for CI-supervised projects.

Some general considerations must be made while interpreting the above results. First, it must be acknowledged that the differences in rating scores are relatively small. Nevertheless, the analysis shows that directly supervised projects have performed better, compared with the CI-supervised ones, according to most indicators considered in this evaluation. Additionally, direct supervision has contributed to enhanced results in furthering IFAD's broader objectives, such as policy dialogue, partnership-building and knowledge management. The evaluation also notes that the results of direct supervision could have been even greater had the pilot programme been implemented under more favourable conditions, for example if the CPM workload had been appropriately prioritized or if adequate monitoring and assessment systems had been put in place. However, in analysing the results, one must also take into account the time allocated to direct supervision. More than half the CPMs involved in DSPP said they spend up to double the time on direct supervision than on other projects in the portfolio. There are also issues of cost, which will be discussed later.

Quality assurance. IFAD lacks a continuous quality assurance system for supervision, which would have allowed the Fund to meet more fully the objectives of the pilot programme. As a result, there was limited quality assurance in direct supervision inputs and processes (e.g. mission terms of reference, composition and duration in the field) and in deliverables such as supervision reports. Other IFIs, in particular the African Development Bank (AfDB) and the World Bank, also have quality assurance mechanisms that allow for periodic assessment of supervision activities (e.g. once every two years). Through these quality assurance mechanisms, IFIs are able to take a holistic view of the supervision function and suggest corresponding systemic improvements across the organization. Such quality assurance mechanisms in the aforementioned IFIs are located outside the operations departments. The Inter-American Development Bank (IDB) is in the process of setting up a similar quality assurance system. In addition, IFIs generally have quality assurance mechanisms for supervision built into the operations divisions (e.g. at the African Development Bank, peer reviews are used for quality assurance).

Reporting to the Executive Board. Management has not complied with two key commitments in terms of reporting. First, no MTR of the pilot programme was undertaken in the last quarter of 2000/first quarter of 2001, as had been decided by the Governing Council. The reasons for this are neither evident nor documented. Second, although the Fund has provided progress reports on direct supervision to the Board on an annual basis, these have focused mainly on the deployment of inputs and processes in implementing direct supervision activities. The evaluation notes that, at the same time, the Executive Board could have exercised better oversight of implementation of the action plan and could have demanded more analytical information from management on the development effectiveness of the DSPP, in particular on the costs, results and lessons learned of the pilot programme.¹⁷

Operating environment for direct supervision. CPMs did not benefit from a favourable operating environment in which to manage the increasing workload caused by the introduction of direct supervision. More specifically, training was not provided at the outset of the pilot programme, nor were CPMs given any particular recognition or incentives. They accepted the additional responsibilities for direct supervision, which they undertook without managerial support for any reprioritization of their existing workload. Last, but equally important, management engagement appeared to diminish gradually after approval by the Governing Council of the DSPP, as demonstrated, for example, by the non-implementation of three key activities¹⁸ included in the action plan.

Learning and knowledge management

By participating in direct supervision activities, CPMs have acquired first-hand experience of the task of supervision and a better appreciation of the issues related to rural poverty reduction. The benefits of their experience are evident, as some CPMs have put their newly acquired knowledge to use in designing new projects and implementing ongoing ones. Still, supervision reports (of both direct supervision and CIs) focus more on implementation issues and less on lessons learned from supervision processes or broad rural development issues. This has led to missed opportunities to incorporate learning into project design across the institution and to improve the supervision activities of cooperating institutions in general. Nonetheless, in spite of the benefits to individual CPMs, there is a need to institutionalize this knowledge and establish systems/platforms through which CPMs involved in direct supervision could share their overall experiences among themselves and within the Fund. Some exchanges have occurred at the CPM level in an informal, unstructured manner. But, overall, institutional support to capture and channel learning from direct supervision to IFAD staff has been inadequate.

IFAD also lacks an adequate reporting and feedback mechanism at the country level: its current system for learning does not allow governments and other development organizations to become familiar with its successful innovative approaches and to learn from its experiences. This needs to be addressed in the light of the Fund's strategic objective of seeking partnerships for replicating and scaling up the activities it finances, as well as of the proposed new operating model. OE recognizes that knowledge management is an institution-wide concern and that direct supervision is only one of the components of the Fund's learning system. However, the unique opportunity for the institution as a whole to learn from the direct supervision pilot programme to enhance IFAD operations has been largely missed.

Efficiency of direct supervision

Based on the calculations of the evaluation, the average cost per year per project of direct supervision is around USD 93 000 as compared with USD 61 000 for CI-supervised projects in the control group. In this regard, it should be noted that the costs incurred go beyond the expectation of the Governing Council, which had decided that in implementing direct supervision activities "there would be no cost increase" to IFAD.

However, there are some points related to the average CI cost that merit being highlighted. First, there is quite a variation in costs across the different CIs. Hence, the average cost for CI supervision calculated above does not reflect the costs that all CIs charge. For instance, supervision through some CIs (e.g. the World Bank) costs over USD 100 000.¹⁹ The current overall cost of supervision through UNOPS (the CI with the greatest number of projects under supervision) is around USD 79 000 – see Chapter V for more data on costs. Moreover, according to the evaluation, the current cost attributed to UNOPS needs to be increased if the Fund is to expect them to provide enhanced quality services.²⁰ Next, there are additional costs of CI supervision that are difficult to identify and have thus not been included in calculating the total cost of CI supervision. For instance, the Belgian Survival Fund has an annual allocation for CI supervision in its administrative budget, recorded outside IFAD's accounting system. Along similar lines, more recently, IFAD's Asia and the Pacific Division provided additional funds to the UNOPS Asia Office to augment its supervision-related activities. These and other such costs are not included in the average of USD 61 000 calculated for CI supervision. Finally, it should also be noted that the cost to IFAD for direct supervision includes the fielding of around two supervision missions per year to each project for longer durations than those of the average CI supervision activities.

Furthermore, the evaluation underlines that the longer implementation period of projects in the control group as compared with the directly-supervised projects has administrative cost implications for IFAD, which would raise the overall costs of supervision by CIs and which need to be considered.

The evaluation also argues that there are possibilities of reducing the costs of direct supervision to some extent. For instance, this could be achieved by making use of competent national entities to discharge the fiduciary responsibilities involved in supervision and thus enhancing the role of government in implementation support activities. Greater use of local consultants could also contribute to cost savings. Finally, as CPMs and the other IFAD staff involved acquire the necessary competencies and gain more experience in direct supervision, it is fair to assume that the overall time invested in the associated tasks is likely to reduce to some extent. This will have a corresponding effect on the staff costs component, leading to a reduction in the overall costs of direct supervision.

While the cost of direct supervision may still be higher than that of CI supervision, according to the evaluation it is paramount to assess the cost together with the corresponding benefits to the Fund. The analysis of the evaluation illustrates that direct supervision has contributed to better development effectiveness and has allowed the Fund to further its catalytic objectives of innovation, policy dialogue and partnership development. With regard to the latter point, partners at the country level expressed their preference for building partnerships directly with IFAD rather then managing such processes through proxy institutions such as CIs. Moreover, in the context of knowledge management, although the CPMs involved in direct supervision have acquired better understanding of implementation matters and despite the fact that knowledge from the DSPP has not been properly institutionalized, the evaluation observes that using a CI for supervision introduces an extra layer into the already feeble learning loop of the Fund. On a similar note, the knowledge that the staff or consultants of CIs have acquired by undertaking supervision on behalf of IFAD is largely lost to the Fund. Finally, the common opinion of governments and all other partners at the country level, who clearly favour IFAD's direct involvement in supervision activities, must also be given due consideration.

Supervision systems and experiences of other International Financial Institutions

By and large, the AfDB, the Asian Development Bank (AsDB), the IDB and the World Bank have similar supervision systems, with processes and details applied variously to fit their institutional structures. In the IFIs, generally, supervision is not limited to official missions and formal reports, rather it is a continuing and flexible process, specific to the needs of the particular operation and intended to foster a close partnership among an IFI, its borrower, and the implementing agencies. The planning of supervision is done carefully during the design phase of the project cycle, when appropriate resources and arrangements are put in place to facilitate supervision. The supervision process normally starts after the project is approved and ends when the last disbursement is completed and the project completion report is prepared.

At the Inter-American Development Bank, supervision is largely delegated to the country offices, with participation of staff from headquarters as required. At the Asian Development Bank and the World Bank, headquarters and country offices share responsibility for supervision. That is, in some cases, supervision responsibilities are entirely delegated to the country offices, especially in those countries in which offices have staff with the required sectoral know-how. Under such arrangements, selected staff at headquarters are sometimes asked to join supervision missions. In other cases, the "task managers", if based at headquarters, retain responsibility for supervision and, in turn, involve staff at their country offices in supervision missions. At the African Development Bank, so far, task managers based at headquarters have full responsibility for project supervision. However, with the establishment of 26 country offices by the end of 2006, AfDB expects in-country staff to be involved in one way or another in project supervision and related follow-up, as well as in portfolio management issues. One important aspect of supervision by these IFIs is that certain supervision functions are being increasingly located in their country offices. For example, the review of documents related to fiduciary aspects (such as bidding proposals and accounting matters) is largely handled by the country offices. It is important to realize, however, that the degree to which such field office staff can make decisions varies according to the delegation provided by the responsible task managers. Finally, some IFIs have well-established quality assurance mechanisms for supervision and others are rapidly moving in the same direction.

Conclusions

A consistent trend in the overall evaluation analysis demonstrates that, compared with CI supervision, direct supervision has greater potential to contribute to better development effectiveness at the project level and, at the same time, allows greater attention to IFAD's broader objectives at the country programme level. That is, direct supervision by IFAD can contribute to better and timelier project implementation, which in turn enhances overall results and impact. Moreover, through direct supervision, IFAD has been able to emphasize issues of prime concern such as gender mainstreaming, targeting and the building of grass-roots institutions, which taken together are important elements in ensuring sustainability.

In view of their more frequent and longer presence in countries with direct supervision, CPMs have wider opportunities to advance IFAD's objectives at the country programme level, including policy dialogue and partnership development. CIs do not consider these to be a priority, nor can such activities be effectively advanced through third parties. Although there is no conclusive evidence that new partnerships have resulted from the DSPP, governments and other development partners at the country level have unanimously expressed their deep appreciation of the more frequent contact with CPMs, which has been facilitated by IFAD direct supervision activities. The same partners communicated that they find it more useful to deal directly with IFAD staff rather than with CI representatives. In this regard, for example, partners affirmed that the response rate and follow-up on implementation issues are faster under direct supervision than under supervision by CIs.

Direct supervision has contributed to developing IFAD's knowledgebase. In this regard, in particular, the CPMs responsible for direct supervision have acquired knowledge of supervision processes, project implementation and rural development issues in the countries concerned. This knowledge has enabled them to better design and implement new operations. However, the knowledge gained at the CPM level has not been systematically shared with others, nor has it been institutionalized, which is one of the key shortcomings of the DSPP. The evaluation notes that supervision by CIs also offers possibilities for knowledge generation. However, CI involvement in supervision makes the transmission line of knowledge from the CI to IFAD and the country more cumbersome in the already feeble knowledge systems of the Fund.

The evaluation concludes that direct supervision allows CPMs to strengthen country-level coordination both within the context of IFAD operations and with the development community at large. It also facilitates the strengthening of existing IFAD-funded programmes and the identification of new programmes and cofinancing opportunities, which are mostly available at the country level, given that the majority of our international and bilateral partners have delegated an increasing amount of authority to their country representatives.

Unlike most other IFIs, IFAD lacks a quality assurance system for direct supervision. As a result, the DSPP was approached and implemented in a variety of ways, based on the perception and understanding of individual CPMs. Compliance with the minimum supervision requirements and the direct supervision guidelines was also not monitored. In conclusion, both continuous and periodic quality assurance systems are fundamental if direct supervision activities are to be expanded.

At face value, the average cost of direct supervision (USD 93 300) is higher than the average cost of supervision by CIs (USD 61 461)²¹. However, the evaluation argues that cost should not be seen in isolation from the benefits the DSPP has evidenced. Moreover, in discussions with UNOPS (the main IFAD CI), it is clear that the costs to IFAD for supervision by UNOPS need to be increased if they are to deliver the type and quality of service IFAD requires in the future. In parallel, the evaluation notes that there is the potential for efficiency gains in direct supervision if, for example, the fiduciary responsibilities related to supervision are entrusted to competent national entities and greater use is made of local consultants for implementation support activities.

The evaluation notes that management's interest has appeared to gradually diminish following approval of the DSPP by the Governing Council. This is illustrated by the fact that management did not fully implement all decisions of the Governing Council. For instance, it did not undertake a mid-term review of the DSPP as required. Neither did it establish an integrated, analytical accounting system to track costs of the DSPP. Nor did it set up a monitoring and assessment system to measure the performance and impact of direct supervision. According to the evaluation, however, neither did the Executive Board exercise adequate oversight to ensure that IFAD management would fulfil all its commitments under the DSPP. The evaluation believes that the outcome of the DSPP would have been even more significant had all the requirements laid down by the Governing Council been implemented.

The evaluation reveals that not all concerned have the same understanding of the notion of supervision. In fact, there is often confusion as to what constitutes supervision missions, implementation support, follow-up activities, fiduciary responsibilities and so on. There is also a lack of clarity on the roles and responsibilities of IFAD, CIs, project staff and government authorities. Additionally, it is worth noting that many partners at the country level felt that the term ‘supervision' – when applied to the implementation aspects of projects rather than to fiduciary aspects – has a paternalistic undertone and they felt uneasy with its continued application. Supervision in that sense reflects a top-down, non-participatory approach to the function, which is inconsistent with the Fund's objectives of promoting ownership and partnership with governments and other institutions.

Recommendations

It is important to state that the evaluation's recommendations have taken into consideration the relevant recommendations contained in the ESM report. The evaluation's five key recommendations, given below, are mutually reinforcing. The evaluation recommends that they all be implemented fully in order to ensure that the desired impact is achieved in IFAD's future efforts in the area of supervision and implementation support.

Recommendation one: definition of supervision

The evaluation recommends that the concept of ‘supervision' as used by IFAD be divided into two distinct operational parts: (i) supervision of fiduciary aspects, including aspects related to procurement review, disbursement processing and compliance with financial and auditing requirements; and (ii) support to programme and project implementation.²² This would include, for example, the organization of periodic ‘implementation support' missions and related follow-up; an assessment of the achievement of programme/project objectives and assistance in identifying remedial solutions for implementation challenges, based on interaction and dialogue with project authorities and other partners at the country and project level; and the provision of guidance in preparing the annual work plans and budgets. It would also include oversight of project and programme implementation, for example, in terms of monitoring the achievement of physical targets.

Recommendation two: develop a comprehensive supervision²³ and implementation support policy for IFAD

The Fund should develop a specific overall supervision and implementation support policy for its operations. The policy should reflect the following elements.

Supervision of fiduciary aspects. IFAD should be allowed to decide, on a case by case basis, whether to subcontract a competent national, regional or international entity to perform such functions. Special efforts should be devoted to engaging national entities, as this would have the double effect of building local institutional capacity and reducing costs. The utmost attention should be given to ensuring that there is no conflict of interest between the prospective national entity and the IFAD operation under consideration. In a few and very specific circumstances, IFAD might consider undertaking the supervision of fiduciary aspects itself.

Implementation support. The evaluation recommends that the policy state explicitly that: (i) IFAD should be made responsible for providing direct implementation support to all its operations globally. In this regard, it is important to specify the role of the CPMs, who could either be intensively involved themselves as "implementation backstoppers" or act as implementation-support task managers with more attention to process management, which would also require a degree of direct involvement in activities. In both cases, CPMs would ultimately be responsible for the process, content and outputs of implementation support activities; (ii) such support would cover all aspects of IFAD country programmes, both at the project level and beyond, such as policy dialogue and partnership strengthening; and (iii) the role of partner governments would be specified and given due emphasis, which would contribute to building greater ownership and local capacities, as well as reducing costs.

The above would result in new responsibilities for the Programme Management Department (PMD), which would require the allocation of additional staff and financial resources as well as new competencies and skills. It is fundamental that the additional resource requirements for the implementation of the new policy be clearly articulated in a comprehensive and explicit manner by IFAD. This would require a detailed cost analysis, particularly of the elements in paragraphs 65-66, as well as an assessment of the skills and competency of current CPMs. Based on the aforementioned analysis, IFAD would need to develop a proposal to meet the cost deficits in implementing the new policy. It would also need to develop a plan for enhancing the ability of CPMs to meet the specific requirements of the new policy, recognizing that it may not be possible to enhance these skills and competencies in all cases. Until the required level of financial resources would be made available to the Fund and additional PMD staff recruited and their competencies and skills developed and upgraded, IFAD might consider a phased approach to expanding implementation support in all operations.

The success of the new policy would also be determined by the support provided by IFAD management and the conducive environment it creates for the purpose. For example, management would need to: (i) ensure that appropriate opportunities are introduced for periodic staff training; (ii) establish an incentive framework and platforms for the sharing of knowledge acquired by CPMs; and (iii) allocate the necessary time for reviewing implementation experience under the policy. The Board should also play a proactive role in exercising oversight of implementation of the new policy and in approving administrative budgets for the purpose.

The policy should be evaluable, and in particular include a roll-out and implementation plan, with performance indicators that can be monitored periodically.

A chapter on supervision and implementation support should be included on a standing basis in IFAD's annual Portfolio Performance Report. The chapter would provide an analytical account of the opportunities and challenges in the area, and identify key lessons learned. Moreover, it would provide an indication of the ongoing operational measures introduced by IFAD to address emerging issues.

The undertaking of supervision of fiduciary dimensions and implementation support would require revisiting Article 7, Section 2(g) of the Agreement Establishing IFAD.

Other integral aspects that the policy should consider are contained in recommendations three to five.

Recommendation three: supervision and implementation support in the framework of the COSOP

The evaluation recommends development of an overall approach to supervision and implementation support during the preparation of COSOPs. This would take into account the need to supervise the fiduciary dimensions of all operations, and the provision of implementation support to the country programme, including areas such as policy dialogue, partnership-building and knowledge management, in addition to the support traditionally provided to projects. The COSOP would lay the provisions for the need to develop an annual supervision and implementation support plan for each country, indicating the specific objectives, human and financial resource allocations and expected results. Each PMD regional divisional would set up an electronic monitoring, assessment and reporting system, which would serve as a management tool to track the implementation progress of the plans and to flag issues requiring more immediate follow-up. A template should be developed by PMD for the section on supervision and implementation support that would be included in the COSOPs.

Recommendation four: quality assurance system

Within the framework of an overall enhanced quality assurance system at IFAD, there is a need to introduce quality assurance mechanisms for the supervision of fiduciary dimensions and implementation support activities. The evaluation recommends that IFAD establish a management review committee within PMD, which would review supervision and implementation support activities, results and related operational issues. Quality assurance, as well, needs to be strengthened in the PMD divisions. In this regard, semi-annual reviews of supervision and implementation support activities should be undertaken at the divisional level. Summaries of the discussions at these meetings should be circulated to all PMD divisions. Moreover, IFAD should build on the experience of other IFIs (in particular of the Quality Assurance Group at the World Bank) to establish an IFAD-specific quality assurance group, which would review aspects of supervision and implementation support, in addition to any other aspects related to implementation of the COSOP and its components. The Fund would need to thoroughly reflect upon the most appropriate location within IFAD's organizational structure for such a group, which would ensure the most objective and independent review possible of its supervision and implementation support efforts. The introduction of such a quality assurance group should take into consideration the mandates and performance of existing quality control systems within IFAD, such as the project development team, Technical Review Committee and Operational Strategy and Policy Guidance Committee.

Recommendation five: learning and knowledge management

Incentives should be provided to staff to encourage sharing of the knowledge they acquire. For example, time needs to be carved out in CPMs' workload for sharing the knowledge they generate through supervision and implementation support activities. The documenting and sharing of knowledge should be included as an indicator in assessing the annual performance of CPMs. Specific instruments need to be established to facilitate learning and knowledge-sharing. In particular, time should be reserved on a standing basis in the CPM forum for discussing issues and sharing knowledge generated through supervision-cum-implementation support activities. Each project mid-term review and project completion report should include a specific treatment of supervision and implementation support issues, as should all evaluations undertaken by OE. The project and country status reports should be reformatted to include a narrative section on supervision and implementation support, and ratings must be included in all cases. Other instruments should be introduced, such as peer reviews at the PMD divisional level in relation to implementation support activities.

The monitoring and evaluation systems at the project level need significant strengthening if they are to contribute effectively to learning. Moreover, in line with the new operating model, it is necessary to assist in the development of integrated monitoring and evaluation systems at the country level. This would not only facilitate monitoring and sharing of experience across the entire project/programme portfolio, but would also allow for tracking the implementation of the broader objectives of IFAD country programmes, such as policy dialogue and partnership-building.

IFAD should build on the experiences of other international financial institutions, and make more comprehensive use of information technology for knowledge management purposes in relation to supervision and implementation support. In this regard, it is recommended that the existing PPMS be expanded so that it can carry updated summaries of supervision and implementation support activities at all times. An enhanced PPMS should accordingly be made accessible to external partners through the IFAD internet with immediate effect. Last but not least, an integrated, analytical accounting system should be developed through close cooperation between PMD and the Office of the Controller. This would allow the monitoring and analysis of all costs related to supervision-cum-implementation support, including staff time.

---

1/ The resolution will cease to be effective in June 2006, as the last project (India) became effective in June 2001.

2/ For the purpose of this evaluation, the term ‘development effectiveness' encompasses the extent to which the DSPP's overall objectives have been met, the efficiency in implementing the pilot programme and the contribution of direct supervision in improving project implementation and potential project impact.

3/ For definitions of the criteria, see Methodological Framework for Project Evaluation (document EC 2003/34/W.P.3).

4/ These impact indicators were developed and presented by IFAD management to the Executive Board in April 1999 (see Box 1 of the present document and document EB 99/66/R.10/Rev.1).

5/ In Armenia, the Gambia and Uganda.

6/ In Bangladesh, Benin and Indonesia.

7/ In Armenia and India.

8/ In Bangladesh, Guinea, Indonesia and Peru.

9/ The role of the CLP was to provide comments and input at several key stages in the evaluation process (for a definition of the CLP, see paragraph 33 in the IFAD Evaluation Policy, document EB 2003/78/R.17/Rev.1). The members of the CLP for the DSPP evaluation were: Mr Nigel Brett, Mr Jim Carruthers, Mr Pablo Glikman, Mr Shyam Khadka, Mr Luciano Lavizzari, Mr John McGhie, Mr Ashwani Muthoo, Ms Rasha Omar, Mr Mohamed Tounessi and Mr Joseph Yayock.

10/ Mr Hans Wyss, former Director of Operations at the World Bank.

11/ Hence field work was undertaken in 13 countries of the DSPP (Gaza and the West Bank and Zimbabwe were excluded).

12/ The action plan can be seen in its original format in Appendix II.

13/ In December 2003, the Executive Board approved the field presence pilot programme in 15 countries globally.

14/ The rating scale used is from 1 to 4, where 1=negligible, 2=modest, 3=substantial and 4=high.

15/ The list of impact indicators may be seen in Chapter II, Box 1.

16/ The composite rating is the average of the six impact domains and the two overarching factors.

17/ In 1997 the GC had, indeed, decided that "progress, lessons learned and results of the test would be reported annually to the Executive Board" – see document GC 20/L.10/Add.1.

18/ Items 6, 7 and 8 under Recommendatio 5 in the action plan – see Appendix II.

19/ A study by the World Bank's Quality Assurance Group in 2003 concluded that, on average, from USD 100 000 to USD 125 000 should be allocated per year for supervision of community-driven development projects.

20/ The IEE report highlighted the same issue – see paragraph 18 on page 16 (document EB 2005/84/R.2).

21/ Although the actual cost per project per year for supervision by UNOPS, the main IFAD CI, is around USD 79 000.

22/ It could also be called ‘implementation support'.

23/ The term ‘supervision', from this point onwards in the document, means ‘supervision of the fiduciary aspects' related to IFAD financing.