The Programme for Strengthening the Regional Capacity for Evaluation of Rural Poverty-Alleviation Projects in Latin America and the Caribbean (PREVAL) covers all countries in Latin America and the Caribbean (LAC). In addition, some programme activities – notably those related to the use of electronic communications, such as electronic workshops and PREVAL's homepage – have reached users in Africa, Europe and the United States.

The programme was designed to strengthen institutions by increasing the availability of professionals trained in the monitoring and evaluation (M&E) of rural development projects aimed at poverty alleviation in the LAC region. Designated direct beneficiaries were professionals at public and private-sector institutions working in the M&E of rural development projects, such as those funded by IFAD and other international organizations, government agencies, non-governmental organizations (NGOs), universities and consulting firms. The programme was also expected to benefit the projects themselves because better-trained professionals would carry out better-quality evaluations, thus generating useful lessons for the design and implementation of new projects.

The general objective of PREVAL was to develop an institutional capacity in LAC for the evaluation of rural development projects. Specific objectives were as follows: (i) to develop a network of regionally based professionals trained in the evaluation of rural development projects and the design of M&E systems; (ii) to strengthen the evaluation capacity of the executing agency (the Inter-American Institute for Cooperation on Agriculture (IICA)) and other regional, subregional and national institutions; (iii) to generate reference and training materials for professionals working in the evaluation of rural development projects; and (iv) to develop a knowledge base focused on the evaluation experience in countries of LAC.

4. It was expected that PREVAL would pursue the above objectives through three project components:

(i) Training, which was to be the main programme component, would consist in the preparation of training materials and the organization and implementation of courses and workshops.

(ii) Applied research would consist in case studies that should be focused on the experience of projects undertaken by IFAD and other organizations and should seek to generate lessons related to the evaluation of rural development projects, particularly methodologies to evaluate rural extension and rural financial services.

(iii) Dissemination of information would include the wide distribution of reports, case studies and training materials elaborated by PREVAL.

5. Initially, participatory evaluation was also included as a programme component, but it was subsequently excluded as a specific activity. (While the Report and Recommendation of the President to the Executive Board on a proposed technical assistance grant for PREVAL defined "participatory evaluation" as one of four components, the grant agreement signed between IFAD and IICA eliminated it as a component, defining it as an activity that cut across all the other ones).

6. PREVAL was expected to have the following effects: (i) it would provide training opportunities to professionals working in the M&E of rural development projects; (ii) it would facilitate and strengthen the exchange of information among such professionals; (iii) it would generate training materials based on the experience of rural development projects in LAC; and (iv) it would improve the evaluation capacity of the recipient's institution.

7. The design of PREVAL was based on the following assumptions:

(i) The programme would benefit rural development projects carried out by IFAD and other institutions by increasing the supply of well-trained professionals. PREVAL's activities would therefore be targeted to the needs of professionals. It would not, however, respond to the specific demands of rural development projects but would seek to incorporate issues arising from these projects in its training programmes.

(ii) Training activities would be based on the concrete experience of rural development projects in LAC. Thus, applied research would be a key task for PREVAL, as it would provide the basis for training materials.

(iii) The recipient institution would be interested in developing its evaluation capacity, and it had the financial and organizational resources to make a positive contribution during the implementation of the programme.

8. Electronic communications (the Internet and e-mail) expanded greatly during the implementation period. In addition, shortly before PREVAL was launched, FIDAMERICA - the Internet-based system of information exchange for IFAD programmes throughout Latin America (FIDAMERICA was funded through technical assistance grant agreement 310-RIMISP.) - had trained managers and staff from IFAD projects in LAC on the use of e-mail as well as the Internet and its application to rural development problems.

9. A new administration of IICA made substantial organizational changes during the implementation of PREVAL. These changes were aimed at reducing expenditures through the decentralization of tasks and the reduction of technical staff. In addition, member countries of IICA established a policy of zero growth in their contributions, aggravating the financial situation of the institution. These changes negatively affected the ability of IICA to work with PREVAL.

10. Following an initial design period from 1992 to 1994, PREVAL was approved by IFAD's Executive Board in April 1995. The programme's commencement date was December of 1995 and its expected duration was three years. However, since IICA only established the programme coordination unit (PCU) in August 1996, the completion date of the programme slid to 31 August 1999.

11. PREVAL implemented the following activities during its 31 months of activity:

Production and distribution of papers and other printed materials. PREVAL produced and distributed a number of publications. Some provided general information about the programme and about courses and workshops organized both by the programme and other institutions. Others were more specifically directed to professionals and provided practical information about methodologies, for instance relating to the evaluation of gender and training issues or the design, implementation and evaluation of computer information systems for M&E.

Organization of a specialized documentation centre. PREVAL created a documentation centre specialized in M&E, located in San José, Costa Rica. Maintaining an updated documentation centre implied constant efforts to search for, purchase and classify publications relating to M&E. The documentation centre had 652 publications by the time of this interim evaluation.

Creation and administration of an electronic network of professionals and institutions. PREVAL created an electronic network in June 1997 in a joint effort with FIDAMERICA. The network soon became PREVAL's most popular activity, attracting 211 members by February 1999. Of the members, 29 (13.7%) were professionals working in IFAD programmes and projects in LAC - a significant number considering that IFAD has 39 projects in LAC and only 31 of these have electronic mail services. The electronic network made possible a wide range of activities and services focused on the M&E of rural development projects, such as the dissemination of information about training events, literature or employment opportunities; the exchange of information among professionals and institutions; the free reproduction and delivery of technical papers; and the elaboration of a database of specialized consultants. In addition, PREVAL organized two electronic workshops. The first workshop, in September/ October 1997, attracted 172 participants; the second one, in November/December 1998, 318 participants (25 of whom from IFAD projects and programmes). Both workshops served to discuss theories and methodologies of M&E. Finally, the programme established two Web pages, one of which was included on IICA's Website and the other one on FIDAMERICA's.

Organization of courses and workshops for IFAD projects. Courses and workshops were designed for professionals working in M&E units of IFAD projects and included: (a) a workshop in Antigua, Guatemala, organized jointly with the Regional Unit for Technical Assistance (RUTA) - an IFAD-supported institution that provides technical assistance to projects in Central America - to discuss the experiences of different IFAD projects; and (b) a training programme organized jointly with the Southern Cone Training Programme in Rural Development (PROCASUR). (PROCASUR has since become the Corporation for Regional Rural Development Training). The latter included three training courses of five days each held at four-month intervals. Each participant had to complete a workshop project focused on the design or reformulation of the M&E system of his or her project. The participant was given individual assistance in the implementation of this project by a specialized consultant who visited each participating project.

Case studies. PREVAL completed three case studies focused on the experience of IFAD projects related to the following issues: (a) M&E systems; (b) the evaluation of gender issues; and (c) the evaluation of training. The results of these case studies were disseminated through the programme's electronic workshops and publications.

Impacts

12. PREVAL carried out most of the activities envisaged at the design stage within the expected time frame. Moreover, the programme implemented activities not foreseen during the design stage, notably the organization and administration of an electronic network and the implementation of activities based on it, such as electronic workshops. The programme had positive impacts in the short term, and it is likely to have positive impacts in the medium and long term, especially as a result of its training activities. In addition, PREVAL developed valuable experience in an area that has been given only scarce attention by other international organizations, generating important lessons for the future.

13. PREVAL attracted a great number of users. Most were practitioners with considerable experience in rural development project work but limited knowledge of M&E issues. Most users stated that they had had great expectations when they joined the programme because of the very limited availability of other training opportunities in M&E in LAC. The great majority had a very positive opinion of the programme's activities, with all surveyed users stressing that PREVAL had adequately met or more than met their initial expectations.

14. The main impacts of the programme were the following:

(i) Creation of a network of professionals and institutions specialized in M&E, enabling users to have greater access to literature and to information about employment opportunities and other professionals and institutions. Of the users surveyed, 92% stressed that PREVAL had contributed to improving their access to literature and information while 76.5% mentioned improved contacts with professionals and institutions.

(ii) Dissemination of knowledge about M&E. Some 518 of the users surveyed found that PREVAL had helped them acquire new knowledge about M&E theories and methodologies, and 74% stated that they had incorporated this knowledge into their everyday work.

(iii) Higher responsibility levels of users. One fourth of the users surveyed claimed to have gained higher levels of responsibility within their institutions as a result of their participation in programme activities. This may be considered a high proportion, given the relatively short period of implementation.

(iv) Strengthening of institutional networks and the introduction of new M&E methodologies in projects and institutions. Sixty-five per cent of the users surveyed stressed that their participation in PREVAL's activities had improved their institutions' capacities to offer services in the M&E of rural development projects and had created new opportunities in this area. In addition, 41% of the users stated that their participation in PREVAL had prompted their institutions to introduce new M&E methodologies.

Cooperation with other institutions and regional programmes. PREVAL cooperated closely with two IFAD regional programmes: FIDAMERICA (for the electronic network and workshops) and PROCASUR (for training activities). Thanks to its links with these programmes, PREVAL successfully carried out some tasks that were not included in the initial programme design. In addition, the programme established links with other institutions, such as the Institute of Development Studies of the University of Sussex, United Kingdom - with which PREVAL is jointly editing a booklet - and the Evaluation Centre at Western Michigan University, United States. On the other hand, the programme did not establish cooperative arrangements with universities in Costa Rica and other LAC countries; and its joint activities with RUTA were limited, in spite of the fact that both were located in San José.

The positive effects and impacts mentioned above related mainly to the participation of users in PREVAL's electronic network (defined as "very useful" by the great majority of the users surveyed); the publications prepared and distributed by the programme (defined as "very useful" or "useful" by 92% of the users surveyed); and the training activities for IFAD projects organized with PROCASUR (defined as "very useful" or "useful" by all participants). Training activities were especially successful because they were demand-driven. PREVAL, in fact, provided well-organized and high-quality training on the actual problems encountered by the M&E units of IFAD projects. As a result, participants were able to learn new methods and indicators, which has led (according to the projects surveyed) to substantial improvements in their projects' M&E systems.

16. Some programme activities had lower-than-expected results, notably the Web pages, the consultants' database, and the "electronic" reproduction and dissemination of M&E documents. Only 50% of the surveyed users knew about and had accessed PREVAL's Web pages. Part of the reason for low use was that not all the users had access to Internet services. The vast majority of those who had had access to the Web pages (90%) had found them useful, while 10% did not. The consultants' database had been used by only 36% of the surveyed users; of these, 87.8% had found the service useful, while 12.2% did not. Finally, all the users of the "electronic" reproduction and dissemination of documents had found the service useful. However, 27% of the sample said that they did not know about the service, which may partly be because it has only been available since mid-1998. It must also be recognized that none of these activities had been proposed at the programme design stage.

17. In addition, the case studies carried out by PREVAL also showed lower-than-expected results for several reasons. Some of the consultants hired to implement the case studies had insufficient experience in that type of research. In addition, case studies focused exclusively on IFAD projects, which were not necessarily adequate for the type of issues analysed.

18. Relations with the executing agency. Unfortunately, PREVAL did not succeed in strengthening IICA's institutional capacity in M&E. According to the institution's management, the main reason for this negative result was IICA's organizational reforms during the period of implementation, which caused IICA's divisions to have very little interest in collaborating with PREVAL. It is, however, objectively impossible to determine what would have happened had IICA had a different institutional context during the past three years. In any case, it is a fact that the relationship between PREVAL and IICA was very limited and mainly concerned with the administrative supervision established at the design stage.

Sustainability

19. The programme has transferred skills, promoted the introduction of new methods and indicators in projects and institutions, and created a network of institutions and professionals specialized in the M&E of rural poverty-alleviation projects. In addition, training activities directed at IFAD projects contributed in several cases to improving the quality of M&E systems - for instance, new indicators were incorporated and new systems were designed or pre-existing ones redesigned. Although these impacts are likely to remain whether or not PREVAL continues, efforts are still needed to expand the number of users, particularly in activities such as training-cum-technical assistance, which has had very positive results but has reached only a small number of beneficiaries, especially in relation to the large demand that exists for this kind of training in LAC. In addition, activities aimed at strengthening the network of institutions and professionals (specifically the electronic network) need to continue in order to attract more users and incorporate new services (for example, electronically delivered training courses).

20. The programme has a good potential for reaching financial self-sustainability in the medium and long term. In fact, the programme could charge for several of its services, such as participation in electronic workshops and other training activities and the reproduction and delivery of copies of documents relating to M&E. Thus, a second phase of the programme could begin to test ways of raising revenue and should prepare a definite plan for becoming financially self-sustaining.

Programme Activities

21. PREVAL should prioritize activities that have performed well, notably training-cum-technical assistance and electronic network activities. In contrast, it should decrease or simply discontinue applied research activities, unless the programme is willing to make the effort and commit the financial resources required to carry out this activity successfully. Such an option would imply focusing on targeting practitioners rather than researchers - as was made implicit by the strong emphasis on applied research at project design.

22. Training and electronic network activities could be improved by incorporating services that interviewed users expressly stated they would like PREVAL to provide. In this connection, users frequently mentioned electronically delivered training courses, an exchange programme to visit projects with successful experiences in M&E, and access to summaries of evaluations of IFAD projects.

Objectives for a Second Phase

23. Based on the results of this evaluation, a follow-up programme could have the following features:

(i) It could maintain the general objective of strengthening institutional and professional capacity in LAC to undertake the M&E of poverty-alleviation projects.

(ii) It could not only seek to strengthen the regional "supply" of consultants and institutions able to provide good-quality services but it could also seek to handle specific training requests from IFAD projects.

(iii) The programme could place greater emphasis on training and on the electronic exchange of information (its electronic network, workshops, and so forth), decreasing its applied research activities.

(iv) Financial and institutional self-sustainability should be a crucial objective in the long term. Therefore, a second phase should encourage the introduction of user fees and the formulation of a plan for income-generation, within a stable institutional set-up.

Institutional Arrangements

24. The institutional location of the programme had lower-than-expected results because IICA showed limited interest in PREVAL's work and contributed few of its own resources to the programme. Thus, if a second phase of the programme is approved, IFAD should seriously consider other options in terms of the executing agency. The possible institutional scenarios, which should be carefully analysed, are the following:

(i) Maintain the current institutional arrangements with IICA. This option would require negotiating changes in the agreement to avoid the problems faced during phase one.

(ii) Maintain IICA as the implementing institution, but with PREVAL being based in a country office different from IICA's central offices in San José, Costa Rica. This option would require negotiating legal agreements and institutional mechanisms to prevent pressures being exerted to concentrate PREVAL's activities in the country or subregion in which PREVAL is located.

(iii) Select an NGO as the implementing agency, following the experience of other regional programmes such as FIDAMERICA and PROCASUR. Such an option would require mechanisms similar to the ones mentioned in (ii).

(iv) Select another regional institution that has yet to be identified. This option would require a region-wide negotiation process before a second-phase proposal is finalized.

25. An additional scenario valid for all the above options would be to initiate a second phase with contributions not only from IFAD but also from other institutions such as the World Bank and the Inter-American Development Bank. This option would allow the size and the potential capacity of the programme to be increased in order to have a regional impact. However, this solution may reduce IFAD's influence on the strategies, supervision and evaluation of PREVAL.

26. The following criteria should be taken into consideration when selecting the recipient institution:

(i) With regard to possible locations, the programme should be located in a country with the following features:

It should have telephone and Internet services of very good quality in order to support a well-performing electronic network.

It should have a good supply of professionals specialized in the M&E of rural development projects in order to facilitate the use of short-term consultants.

It should have easy and low-cost access by air in order to facilitate the travel of programme staff within LAC.

It should have institutions working in areas of PREVAL's interest in order to make alliances and joint activities easier.

The subregion should have a high concentration of IFAD projects.

(ii) With respect to the recipient institution:

It should be specialized in activities related to the M&E of rural poverty-alleviation projects.

It should show great interest in PREVAL's activities.

It should manifest the will and the capacity to contribute human and financial resources to the programme.

It should have organizational features that allow the programme to work autonomously and efficiently, which implies that bureaucracy will be kept to a minimum.

27. In addition, the programme should consolidate the co-operation already initiated with institutions and regional programmes such as FIDAMERICA, PROCASUR and RUTA, while undertaking new cooperative agreements with other institutions such as universities and NGOs with similar training programmes.

The organizational structure of the programme - which now consists in basically one coordinator with almost sole responsibility for operationalizing strategic planning, monitoring and evaluation - requires some changes to promote new ideas and to propose solutions to problems as they arise. For instance, a technical committee composed of members of IFAD, the implementing institution, and some other guest expert members could assist the coordinating unit with these functions. For the same purpose, additional human resources could be incorporated at a low cost in order to create more time for the programme coordinator to work on strategic and planning activities. For example, the programme could use one or more associate experts (commonly used in several institutions) to cover some of the programme activities, such as the electronic network.

29. The training model proposed by the "integrated training plan" currently emerges as the most successful one in terms of impact, both on the professional level of the beneficiaries and on the M&E system of the projects and institutions involved in project implementation. Within such a model, the formula "training-cum-technical assistance" was found to be the most valid one for creating a positive and effective role for the M&E units in the management and strategic planning of rural poverty-alleviation projects.

30. A programme that combines a wide range of activities, some of which are based on innovative tools such as electronic networks and workshops, requires an M&E system that incorporates new indicators. Indicators should, for example, pay substantially more attention to the indirect impact the programme has on the performance of the projects and institutions it supports through its activities, rather than concentrating on its direct effects on individual participants.

31. Applied research is an academic activity, even when it focuses on the identification of practical lessons learned from operational experience. In order for a programme that is focused mainly on training to carry out applied research effectively, considerable human and financial resources are required. Therefore, a programme such as PREVAL, which is seeking to carry out many activities with very limited resources and a life span of only three years, should become involved in applied research only when it can reasonably foresee reliable and relevant results within its severe resource constraints.

32. A programme that includes applied research should not be limited to carrying out case studies focused on IFAD projects only. Case studies should focus on successful experiences or interesting innovations that may generate lessons for the design and implementation of project activities and M&E systems, regardless of whether they are or are not funded by IFAD.