Completion evaluation ¹

Introduction

The completion evaluation of the DDSP was converted from an interim to a completion evaluation mainly due to the realignment of the policy of the Government of Uganda towards a sectoral funding approach, which appeared to make a second phase unlikely. The task of the Completion Evaluation mission has been to evaluate the DDSP and to make recommendations concerning IFAD's future strategy and approach in Uganda, with a second phase or follow-up programme as possible options among others. The mission was in Uganda from November 19 to December 10 2004, meeting with stakeholders in the five programme districts and at central level.

IFAD has funded or jointly funded 13 projects in Uganda since 1981, shifting in the 1990s from area-based interventions focusing on agricultural development to multi-component rural development projects co-financed by the Belgium Survival Fund (the DICDPs) and to projects concerned with export products such as cotton and vegetable oil. The three most recent projects constitute a further shift, towards countrywide sectoral approaches. The DDSP was conceived as a follow-up to the rural development projects in the western region and designed to strengthen the process of decentralisation which had started in Uganda in the late 1980s.

The evaluation follows the IFAD methodological framework for Programme Evaluation, which outlines the main evaluation criteria and provides a rating system. The methodology utilized by the mission for impact assessment included: (i) semi-structured interviews with groups and committees; (ii) focus group discussions; (iii) one-to-one interviews with householders and key informants; and (iv) the review of official documents and socio-economic literature. Poverty ranking exercises were also carried out with members of community-based organisations.

Main design features

The DDSP is a USD 21.5 million programme co-financed by the Belgium Survival Fund and Development Cooperation Ireland ² and directly supervised by IFAD. It was declared effective in May 2000. The initial closing date was December 2004, later extended to June 2006. The target area comprised the three western districts of Hoima, Kibaale and Kabarole, with a combined area of 18 000 sq. km. and a population of 1.4 million. These districts were among the poorest in the country, with average earnings less than half the national average and malnutrition still widespread.

The main emphasis of the DDSP was to be on productive activities combined with consolidation of the previous socio-economic interventions. The direct beneficiaries were to be around 50 000 households, with a particular emphasis on women and on children at risk, and a broader focus on poor farming households, fishermen and traders. The associations and committees formed under the DICDPs were to provide the basis for community participation.

The goal of the programme was to alleviate the chronic poverty in the programme area through raising rural incomes, improving health, food security and nutrition, the participation of farmers in commercialised agriculture and the enhancement of local governance. For Kabarole, which had not benefited from community-based development assistance, the programme was also designed to create the necessary institutional frameworks. The programme components were: community development, rural finance, health and nutrition, water and sanitation, rural roads, agricultural development, and management capacity building and institutional strengthening.

The Ministry of Local Government had overall responsibility for the programme and for providing the liaison and secretariat capability through a dedicated unit at central level. To the Inter-Ministerial Policy Committee was given the advisory role on policy and strategy issues. The implementing agencies at field level were the district administrations, with a District Technical Planning Committee acting as the key coordinating body. The Uganda Women's Finance Trust was selected for the implementation of the rural finance component, including the creation and guidance of groups and the management of the credit fund.

Changes during implementation: (i) the division of the former Kabarole into three districts by the creation of the new districts of Kamwenge and Kyenjojo resulted in a one year delay in start-up in the new districts; (ii) the government decision to abolish community contribution in basic health care caused the abandonment of the drug revolving fund and militated against the DDSP approach to sustainability; (iii) the operation of the health and agricultural services was affected by the ceilings on staff recruitment during the restructuring of local government; (iv) the proposed ten-year suspension of Graduated Tax has already had a significant impact on local government revenue flows, and no replacement plan has yet been announced. Design changes during implementation include the setting up of the Bunyoro Toro Development Company (BUTO), a limited company jointly owned by the five districts, to take over the running of the rural finance component.

Summary implementation results

Community development (12% of base costs). The major activities have included the construction of community centres, the training of literacy instructors and Community Development Assistants and the promotion of adult literacy groups, agricultural commodity interest groups and women's savings and credit groups. The component has also assisted in the formation of users' committees for other components. Community centre construction has been satisfactory, and the maintenance committees function well. The main problem for the component is the shortage of staff (in some districts) and dependence on unpaid volunteers.

Rural finance (14%). There is now a Subcounty Integrated Development Association in every subcounty in all districts, as planned. The SIDAs have been able to mobilize over UGSh 400 million in savings and many should soon be able to operate as independent microfinance institutions, without external support. The transition to BUTO management has been smooth due to the retention of trained staff and the intensive training of SIDAs in technical and managerial aspects of savings and credit operations.

Health (8%). Health has the smallest of component budgets, with very small amounts allocated to the new districts, where primary healthcare coverage is still very limited. The major activity has been the construction of nine Level III Health Centres, three each in Hoima and Kibaale and one each elsewhere. The quality of construction is satisfactory, but basic amenities were missing from the original design, notably running water and electric lighting for the maternity units. Training courses have concentrated on village health workers, traditional birth attendants and community health committees.

Water and sanitation (21%). The major investment has been in borehole construction, which is complete in Hoima, Kibaale and Kabarole. Water source committees and hand pump mechanics have been trained for the maintenance of the boreholes. Water tanks and ventilated pit latrines have been installed in primary schools, the latter with handwashing facilities as part of the design, maintained by pupils under the guidance of school health committees. In Kibaale and Hoima, sanitation platforms were provided to households at subsidised prices. Hygiene improvement included the promotion of handwashing, washing racks and rubbish pits.

Rural roads (17%). The largest item of expenditure has been the construction of 1 120 kms of community access roads, with 87% of these concentrated in Hoima and Kibaale. The most successful approach to construction has been the direct hiring of local labour gangs. Considering the low costs and lack of skilled labour, the standards of construction are commendable and community maintenance has generally been effective. The other major investment under the roads component is the ongoing construction of two motorable roads to Lake Albert.

Agricultural development (14%). Activities included the distribution of improved crop varieties (especially for bananas and coffee) to progressive farmers for multiplication and further distribution, the setting up of beekeeping centres, the establishment of fish fry ponds and the distribution of improved livestock. Over 20 study tours to other districts were organized and training offered in animal traction, fish farming, farming as a business, and crop and animal husbandry. The hiring of agricultural management specialists has resulted in advances in market orientation and farm record keeping, especially in Kibaale and Hoima, and beginnings have been made in linking farmers with the private sector.

Capacity building and institutional strengthening (14%). District-level staff are proficient in standard computer applications and there has been extensive training on the logframe approach to monitoring. Procurement delays have been reduced to a standard two months. Subcounty and parish officials have received training in record keeping and planning, and timely workplans and budgets are now produced at parish level. Eighty-two Parish Headquarters have been constructed for use in revenue collection and as community centres but implementation has been slow.

M&E and accounting. Despite significant investments, the M&E system is not fully functional and a central repository for data is lacking. Roads, Water & Sanitation, and Education all use separate systems. However, the districts are adept at compiling financial records and component-specific data by hand. Accounts are accurately kept and detailed financial statements produced. The continuing delay in counterpart funding –  USD 481 000 at the end of August 2004 – is disrupting accounting procedures and causing uncertainties in payment schedules.

Performance of the programme

There have been significant improvements in capacities, service provision and infrastructure, especially in the older districts where the bulk of the investment has been made. Safe water supplies, the road network and primary health care have been substantially extended and the cost of infrastructure has been kept within reasonable limits. The principle of community participation in development planning has been established. Overall, the DDSP has been a successful programme, with problems related to design rather than implementation.

Imprecise objectives. The programme's stated objectives are general statements of intent rather than measurable and time-oriented objectives, and this has entailed a long search for suitable impact indicators. The ‘participation by farmers in the monetization of the economy' seems especially imprecise and does not target the poorest farmers. 

Failure to target the poorest.  Only the physical infrastructure components have generated benefits for households in all categories. The mission's wealth ranking exercises indicate that SIDA and farmers' group membership is dominated by the ‘not so poor'. The majority of the ‘poor' and all the ‘poorest' are unable to meet the collateral requirements of SIDA loans, which are unsuitable for agriculture. Activities under the agricultural component are mostly targeted at ‘the active poor' on the grounds that the poorest lack the basic means of production.

Decentralisation issues.  The DDSP has been instrumental in promoting a process of participatory planning whereby community requests are progressively screened at higher levels. However, the long delays and lack of government response tend to generate a sense of powerlessness and frustration. There are clear limits to decentralisation, including the decisions necessarily made by technical departments, the capacities of village and parish development committees and, above all, the fact that without financial decentralisation there can be no meaningful delegation of decision making.

Users' committees. Most water sources showed evidence of neglect. Less than half of the committees interviewed had collected funds for O&M and over a quarter of beneficiaries viewed the water sources as government facilities. In many cases, there had been inadequate preparation before construction of facilities, lack of training  and lack of follow up. As for community health committees, there seemed to be little clear understanding of their function and the mission assessed them as unsustainable. The situation in the road component seems better, with a widespread understanding that the responsibility for maintenance rests with the community.

Infrastructural efficiency. The cost per kilometre of roadbuilding was economical for labour-based schemes and cost-benefit ratios were very satisfactory for health centre construction. Under the water component, however, boreholes – the most expensive solution – were normally preferred even when cheaper alternatives existed. Some doubt hangs over the usefulness of the parish revenue offices in view of the imminent suspension of Graduated Tax. 

High operational costs in rural finance. The Mid-Term Review allocated more for component support than for the revolving credit fund, even though most fixed assets had already been purchased. There are suggestions of extravagance in BUTO's operational budget.³ Despite the purchase of six vehicles for the component, SIDAs are obliged to hire vehicles to transport their monies.

Continuing problems with M&E. The programme M&E system has not repaid the considerable investments in terms of time and money and is still not fully utilised. Each line ministry has its own reporting requirement and corresponding software system, containing mostly quantitative data. The beneficiary evaluation process is complex and time consuming, budgets are inadequate and incentives are lacking for facilitators.

Poverty impact

Water, roads and health. Nearly 200 000 rural people gained access to safe water as a result of DDSP interventions and the average distance to water sources was reduced to reasonable walking distance in most areas. The availability of clean water has had an impact on hygiene and in schools the health of children has improved through the provision of handwashing facilities with the new latrines. The construction of community access roads in the five districts represents an expansion of over 20% within four years. Beneficiaries reported the main impact as cheaper and quicker transportation of farm produce and better access to health centres and schools. The average distance to the nearest health unit has been reduced in all districts. In Hoima, 80% of the population now live within five kms of a health centre and in Kibaale 60%. The number of facility-based births is rising rapidly, especially in the new districts. Hoima and Kabarole have reported substantial reductions in the rate of stunting, underweight and wasting in children under five.

SIDA members. SIDA members reported a marked increase in economic activity and household incomes. Many of them have replaced thatched dwellings with iron–roofed houses and some have piped water. Others have purchased land for commercial or agricultural use and utilised the asset as collateral for obtaining larger loans from commercial institutions. Characteristics such as persistence, calculated risk taking, independence and self-confidence have been developed through training, group cohesion and access to credit for small enterprises. Over 95% of SIDA members reported running profitable enterprises.

Farmers and farming. Increases in crop yields, crop cover and livestock numbers are claimed in the agricultural sector, but convincing statistics were generally unavailable.  Members of farmers' groups reported better access to health and education services and improved production skills through training and study tours. All progressive farmers could point to at least two new skills they had acquired. Farmers' groups are encouraged to save, which has in turn increased their access to credit. Farmer Extension Representatives have emerged as respected community leaders but many farmers outside the groups remain without access to extension services.

Food security. All group interviews, without exception, confirmed the perception of the beneficiaries that the overall incidence of poverty has fallen and that household food sufficiency and family incomes have improved. From the literacy classes, radio programmes and drama groups, there is increased awareness about nutrition, and many literacy students have taken up vegetable growing. Beneficiaries of agricultural programmes also reported improved diets.

Environmental impact. The environmental impact of DDSP activities has been positive, but the mission recommends a rigorous environmental impact assessment as part of the development of the lakeshore of Lake Albert after the construction of motorable link roads.⁴

Gender. Almost all married SIDA women said their husbands were now more respectful, and felt they were more often consulted about issues affecting their lives. A number of SIDA members have taken on leadership positions in the local administrative structure. Women have been active in the majority of users' committees.

Development impetus. The fundamental sustainability issue is whether the districts will be able to maintain the development impetus in the absence of further donor support. The problem is not with ‘hardware' aspects such as roads, boreholes and health centres but with the ‘software' aspects such as capacity building, the effectiveness and dynamism of groups, the sustainability of adult literacy programmes and the concept of agricultural commercialization.

Sustainability of SIDAs. The sustainability of SIDAs seems assured in Hoima and Kibaale, but in Kabarole, Kyenjojo and Kamwenge the associations seem weak in terms of cohesion, ownership, savings, coverage and outreach. There are some doubts about the level of SIDAs' independence from BUTO.

Volunteers. Many programme committees and activities depend on volunteers, especially under the Community Development component, which relies on the voluntary activities of literacy instructors and community ‘change agents'. SIDAs also rely on volunteers, who may not always be forthcoming. Committees under the water, health and roads components all operate on a voluntary basis and will only prove sustainable where the sense of ownership is strong.

Innovation and upscaling. The DDSP was substantially a follow-up to the DICDPs, its chief innovation being the degree and compass of direct support given to local government authorities, probably unique in Uganda. Lesser innovations include: the attempts at cross-sectoral collaboration, especially regarding the overall role of the Community Development Department; the creation by the districts of a limited company (BUTO) designed to manage credit delivery, and the involvement of NGOs in service provision.  

Performance of partners

Performance of IFAD. IFAD is to be commended for the very energetic and effective supervision which in turn has led to the fostering of a direct dialogue with central government. However, the recent Thematic Evaluation complains that ‘the area of dialogue and upscaling has not been seriously explored', for which a full-time country presence is necessary.

The co-financiers. Both the Belgian Survival Fund and Development Cooperation Ireland have a significant history of assistance to the programme area and have taken a close interest in the programme. The BSF is particularly keen on cross-sectoral collaborations and is concerned that the ‘basket-funding' approach will not be able to accommodate this priority.

The Central Government. The Government of Uganda has shown a genuine commitment to decentralisation, but the ten-year suspension of Graduated-Tax without declared alternatives, the abolition of cost-sharing at health centres, the free distribution of seeds and livestock breeds in the agricultural sector, and the temporary hiring freeze in the health service have all tended to undermine the community-based and market oriented approach of the programme.

The Ministry of Local Government, and in particular the key role of its programme liaison office, is highly rated by the districts. The Ministry of Finance, Planning and Economic Development is exploring a harmonized reporting structure to be used in conjunction with the sectorwide ‘basket funding' approach.

The role of NGOs in service provision has been substantial under the programme, extending to the supply of sanitation platforms, the training of water source committees, the provision of extension services and various agricultural training programmes. Most NGOs are reported as performing well, although formal reporting procedures are not always adequate. 

The Bunyoro-Toro rural development Company. It is BUTO's goal to encourage the SIDAs progressively to purchase BUTO shares and ultimately to become majority shareholders, but the feasibility of this commendable plan has not been properly examined. BUTO does not appear to operate within any supervisory framework and the reporting and monitoring systems do not conform with those of the other components. The Memorandum of Understanding according BUTO the necessary legal status for the administration of the RCF is still unsigned.

The SIDAs. The expression ‘SIDA women' conveys an image of self-confidence and entrepreneurial spirit. The Thematic Evaluation's assessment of the SIDAs was that they had had ‘a tremendous impact on women in development'.  BUTO reports all SIDAs as being profitable and the longer-established ones as self-sufficient. SIDAs in Hoima and Kibaale are described as the ‘market leaders' in microfinance. The main drawback is that SIDA membership tends to exclude the very poor.

Conclusions

Overall. The DDSP has been a successful programme. Crucial to its success has been the demanding supervision exercised by IFAD, which has served to make good the weaknesses of the original design – precisely what supervision is supposed to do, and rarely does.

Design issues. The design of DDSP was over-idealistic in respect of ‘bottom-up planning processes'. The limits to the decentralisation of planning decisions were never properly clarified.

Programme effectiveness. The programme was effective in achieving its stated goals except in targeting the poorest households. The implementing agencies considered that for the credit and agriculture components, the ‘active poor' and the progressive farmers comprised the only practicable target groups, and that the ‘trickle-down' effect would carry the benefits to the poor. This forceful argument highlights a dilemma for IFAD, with its brief to target the poorest.

Programme efficiency. Precision about levels of efficiency is problematic in the absence of certain key statistical data. The evidence such as it is does not suggest a costly programme and indicates an efficient use of limited human resources in terms of major investments.

Overall impact. The impact of water sources, roads and health centres is undeniable, the credit component has far exceeded its measurable objectives, and the software packages (training, sensitisation and mobilisation) have had a substantial impact on ‘the culture of governance'.

Recommendations

Continuation of IFAD investment. The mission recommends further investment in all five districts on the grounds that the task, although well-started, is only half-done, and many of the investments may prove unsustainable. This is particularly true for Kamwenge, Kyenjojo and Kabarole, where implementation has been underway for only three and a half years. To remove a chief source of funding for these new districts⁵ should not be done without carefully weighing the consequences. The scale and direction of the new investments will require assessment by district and by component and the conduct of a fresh baseline survey.

Decentralisation of funds. In future projects of this kind, subcounties and parishes should be directly funded by the programme to give substance to the commitment to bottom-up planning.

Poverty focus. The rural finance component of any future project should have as a priority the development of a medium-term loan for agricultural purposes accessible to the very poor.

Strategic issues. In its negotiations with the Government of Uganda concerning future loans, IFAD should be mindful of its unique role and comparative advantages (specifically highlighted in the recent external evaluation of the Fund), neither of which are well-served by contribution to a basket-funded approach.

Counterpart funding delays. The Government should consider setting up monthly or quarterly automated disbursement schedules to avoid obstructive delays in counterpart funding. 

Donor coordination. The harmonisation of reporting systems, urgently required, could be addressed at the monthly meeting of donor representatives in Kampala. 

Community development. Interventions need to be aimed more accurately at the poor and marginalised. To this end, the Community Development staff should conduct training with staff of the agricultural and rural finance components on wealth ranking exercises. Men should be attracted to join literacy classes by marketing them as providing access to extension advice, English and Swahili language skills and economic opportunities.

Rural finance.  Operational funding should be reduced and the revolving credit fund correspondingly increased. Clear guidelines as to the social responsibility of the component should be set out, and appropriate monitoring indicators developed. BUTO should be required to allocate a proportion of credit funds as pro-poor agricultural loans and to follow the same reporting procedures as other implementing agencies.

Health. IFAD should seek clarification on the replacement of nursing assistants before making further investments in the health service. The initial unit cost of new health centres should include necessities such as running water, DC electric power supply, a minimum number of beds and mattresses, and a perimeter fence.

Water and sanitation. As boreholes are expensive, they should be adopted only where there is no other viable alternative. For effective maintenance, communities should be trained before construction starts and a simple guide for committees should be prepared. Initiatives in some districts can be replicated: training by NGOs; associations for hand-pump mechanics; the utilisation of the O&M fund as a revolving credit fund.

Rural roads. The Ministry of Works should decentralise the responsibility for the upkeep of trunk roads to the districts, retaining only a coordinating function. It should also review its commitment to the inefficient contract system for the upkeep of district feeder roads.

Agricultural development. Tools for monitoring component indicators should be developed and relevant stakeholders trained in their use. Efforts should be made to attract poorer farmers to join farmers' groups. The policy of working with NGOs in outreach activities should be extended.

Management. The mission recommends a tolerance level of up to 10% allowing districts to deviate from the original Annual Work Programme and Budget without specific application, subject only to audit controls. Inter-sectoral collaboration would be enhanced by brief monthly meetings between component heads and inter-district collaboration by a bi-annual workshop for the exchange of ideas. The programme should hire an MS Access expert familiar with the existing M&E system to design linkage mechanisms for existing databases and to create a central repository.

---

1/ The composition of the mission was as follows: Roger Norman, Rural Development specialist and Team Leader; Alice Mango, Water and Sanitation specialist; Catherine Komugisha, Rural Finance and Agriculture specialist; Ismet Mustafa, Management specialist; Lea Joensen (IFAD APO) Community Development specialist. The mission was joined by Katharina Kayser, IFAD Evaluation Officer, for its final week in the country.

2 / Formerly Ireland Aid.

3/ BUTO has challenged this statement and some others in this report. See also page 38, footnote 40.

4/ Hoima District has pointed out that an environmental impact assessment was carried out for the road scheme itself.

5/ After the division of the old district of Kabarole, Kabarole itself was also effectively a ‘new' district.