3. THE SIX STEPS OF THE NEW EVALUATION PROCESS ²

The NEP prescribes six steps to be performed in each evaluation. Particular emphasis shall be given to the three main features of the NEP: the Approach Paper, the CLPs and the agreement/understanding at completion point. However, the various activities provided under each of the six steps have an indicative character only, and will be implemented on a case-by-case basis by the EO and the CLP whenever feasible and necessary.

Step #1 – Evaluation Pipeline and OE Annual Programme Development:

OE will engage in discussions with its partnership before the start of each calendar year to identify potential project and programme evaluations, country programme evaluations, thematic studies and related activities that will result in a list of evaluation activities to be undertaken in the following 12 months. It will also establish a pipeline of evaluation activities with a medium-term time frame. This will provide sufficient lead-time for appropriate planning and allocation of OE's human and financial resources. Given PD's proximity to OE, it is natural that we will engage in a more intense dialogue with them during this stage, although subsequently we may take advantage of new communication methods to establish, in collaboration with PMD, a productive exchange with our partners outside Rome as well.

Taking the evaluation pipeline as a starting point, we will develop an annual evaluation work programme, and use this as the basis for further interaction with PD and others. Each evaluation activity we include in our work programme needs to be supported by clear rationale and a compelling justification, including the outcome the partnership expects from the proposed evaluation.

Step #2 - Approach Paper: ³

The Evaluation Officer (EO) is responsible for drafting the Approach Paper. At this stage, s/he may preliminarily define the composition of the CLP⁴, convene it and begin to work in close consultation and collaboration with CLP members in developing and finalising the Approach Paper.

The Approach Paper An Indicative Outline Background & Rationale Background Justification of the Evaluation Expected Focus of the Evaluation Objectives Key questions to be addressed Expected Outcome(s) Partnership involved Core Partnership Broad Partnership Respective roles The Process Phases of the process Work Plan and Budget Budget including Consultant/Resource Persons involved Terms of Reference of consultants. (N.B. see Annex #2 for more detailed description of the Approach Paper)

The Approach Paper provides the overall framework for the evaluation and outlines the issues at stake: justification for the evaluation, expected objectives, expected evaluation outcome, output, and budget estimates. The paper describes the process to be followed from start to finish, with a description of each phase, providing a work plan and schedules for undertaking and completing each phase. For this reason, it is important to consult and agree with CLP members about their availability to participate in the process. The Approach Paper also describes the evaluation methodology (e.g., whether a participatory rural appraisal is to be undertaken, etc.), and defines the specific role of the EO. Whenever required during this step, the EO may want to organise a short field visit to brainstorm with stakeholders and gain insights from the field in finalizing the Approach Paper, in consultation with the CPL.

During step two the EO will do research for information on the experiences and knowledge of IFAD and others on the issues to be addressed by the evaluation. The objective here is to learn and benefit from the work of others, and use these inputs in planning and streamlining the Approach Paper.

Wherever possible the EO will put the Approach Paper to peer review to gain from the experiences and opinions of other OE colleagues before embarking on an evaluation. Lastly, the EO and the CLP will discuss and finalize the Approach Paper.

Step #3 – Selection and Recruitment: Based on the methodology outlined in the Approach Paper, the EO develops a detailed TOR for the individual experts/consultants to be involved in the process. This step involves the selection and recruitment of the evaluation team, with efforts to be made to include local resource persons in the evaluation exercise. The EO will select consultants skilled in communication and committed to learning with others and prepared to work constructively in teams. Their detailed TOR should provide clear and precise instructions, with time frames on their specific role and requested output. During this step, the EO develops the mission's itinerary and makes other administrative arrangements.

Step #4 – Conduct Participatory Evaluations:

The recently-developed OE Vision Statement ⁵ describes the empowerment of the rural poor as the overriding concern of OE. IFAD's participatory evaluation should therefore endeavour to provide special opportunities to enhance the participation of the rural poor in the assessment and improvement of the development services and policies from which they are supposed to benefit. Participatory evaluations are therefore designed with the objective of strengthening the position of the rural poor in their interaction with implementing agencies, governments and IFAD itself.

The type of participatory process, approach and methodology to be used during the evaluation exercise⁶ is defined and agreed upon in the Approach Paper and TORs of the consultants. Pre-evaluation fieldwork may be necessary to gather complementary information through primary and secondary data collection and to discuss and organize the participatory evaluation process.

The evaluation mission itself starts in the field with the first stakeholders' meeting. Depending on the type of evaluation and on the local context, different combinations of methodologies should be used (PRA, key informant interviews, community and district level workshops, etc.). In most cases, however, we will strive to create a bottom-up process of reflection, assessment and dialogue, which allows the target group representatives and other local stakeholders to voice their views and proposals with the project management unit, implementing agencies, government authorities, donors, etc. The evaluation mission's own findings and preliminary conclusions will be presented and discussed in a participatory workshop at the local level which may be followed by a wrap-up meeting in the capital city. The presence of the CPM and the co-operating institution at a followup workshop and/or at the wrap-up meeting would be highly desirable.

Upon return from the field, the evaluation team will organise a debriefing with staff from PMD. The debriefing will be based on the preliminary evaluation findings and conclusions, and documents. The salient points emerging from the discussions would be documented and circulated to the participants and the CLP.

Eventually, the evaluation team drafts the report, including the main text, executive summary, and working documents as annexes, if necessary. The report should be short and packaged in a user-friendly way. It is important that consultants be thoroughly briefed and monitored to ensure that they produce a satisfactory output. The report should not exceed 30-40 pages in length, excluding annexes and the executive summary. Finally, we will use peer review within OE, as and when appropriate to ensure quality standards.

Step #5 – Proposed Recommendations, Lessons Learned and Follow-up:

The role of the CLP at this stage is particularly important. The draft evaluation report should be shared within the CLP as soon as it is ready to enable prompt processing of the evaluation, and to build early ownership in the eventual output. The CLP will prepare an outline of their common understanding on the most important lessons learned and recommendations emerging from the evaluation, as well as identify the follow-up actions required. In addition, the CLP will determine the process required for the evaluation to reach the completion point (step #6). With regard to the latter, the roles and responsibilities of each partner will be specified clearly at the outset by the CLP. The CLP will also define a plan of action to promote dialogue with the larger partnership based on the evaluation results generated⁷, with the objective of promoting the utilisation of the evaluation's most important messages. However, it is understood that some IFAD regional divisions will be more inclined to share draft evaluations with the partnership at large before some degree of internal review than others.

Step #6: The Completion Point: Understanding on outcome:

This stage is considered the completion point of the evaluation process and replaces, together with step #5, the Evaluation Panel, which over time proved not to be an effective forum for the internalisation of knowledge and learning.⁸ At this step, the learning and knowledge generated by the evaluation is officially acknowledged and internalised within IFAD's knowledge systems and by the partnership. This can take the form of an agreement/understanding among the main partners about adoption of the major lessons learned, recommendations produced in the evaluation process and follow-up actions to be undertaken. This understanding will be communicated to the rest of IFAD to ensure dissemination of lessons.

Still, a further round of discussions and negotiations among the partnership may be warranted to reach the completion point. This could take the form of an in-country workshop involving the concerned members of IFAD management, the CLP and the broader partnership. If resources for such a workshop are limited, an in-house workshop could be convened with the same objectives, using mechanisms (e.g., video-conferencing, etc.) to enable an interactive meeting with the members of the CLP and the partnership not present.

The completion point is an agreement to act on evaluation recommendations. The written agreement or understanding illustrates the stakeholders' consensus and commitment to implement evaluation recommendations which are expected to improve project operations, policies, or improve future project design.

A summary of these six steps in OE's New Evaluation Process are presented in diagram form in Annex 1.

² We have made an effort to rethink the evaluation process in abstract from the specific steps traditionally involved in the various types of evaluation instruments (project evaluations, country programme evaluations, etc.). This was done to avoid preconceived schemes. Some of the evaluation steps identified by our review are more relevant for certain evaluations than others (e.g., field work for project evaluations). In such cases we have made explicit reference to the evaluation type concerned.

³ See Annex 3 for more details.

⁴ See Annex 2. The CLP has been developed primarily for Country Programme and Thematic Evaluations. We will, however, experiment with the CLP approach in selected Project Evaluations as well.

⁵ The OE Vision Statement reads as follows: "We want to promote a learning process that deepens our understanding of rural poverty's causes and solutions through enhanced co-operation with our partners. With our partners we want to use this knowledge to develop supportive instruments for the rural poor to empower themselves."

⁶ OE is currently reviewing its experience with participatory methodologies in evaluation in order to identify best practices to be used in the future.

⁷ That is, the agreed upon recommendations, lessons learned and follow-up actions.

⁸ The Evaluation Panel was not convened in the majority of cases: roughly only 71 evaluations, or about 24%, of all evaluations conducted by OE were discussed in the In-house Evaluation Panel since its establishment in 1990. The Evaluation Panel was established on an experimental basis for the following purposes : a) review evaluation reports to improve quality, b) share evaluations lessons in-house in order to for them to be included in the design of new projects; and c)  decision-making on their dissemination. The Evaluation Panel also served to: (d) signal the end of each evaluation. OE believes that the learning function (objectives a and b above) of the evaluation panel in the past was limited to a relatively brief interaction among divisions in a rather formal meeting. Clearly the New Evaluation Process in general and the CPL in particular will go much farther than the evaluation panel in promoting learning through evaluations. As to the formal completion objective (d), this is included in the New Evaluation Approach by the new concept of the agreement among the parties at completion point to be prepared by the CPL. Finally, as to the dissemination of the lessons learned, OE agrees that a specific arrangement that goes beyond EKSYST/IFADEVAL is necessary. This is why OE has decided to develop a dissemination and communication strategy for the results of its evaluation work in 2000.