Value-based evaluation and why it matters

Naidoo, I., Aronsson, I.-L. & Hassnain, H.

Evaluation and values

Evaluations which neglect to consider values are at risk of becoming irrelevant and unused. Human values play a crucial role in shaping individuals’ perspectives, motivations and judgments (Rokeach, 1973). Evaluations that fail to account for these values may overlook important factors that influence human behaviour and decision making, potentially leading to an incomplete or inaccurate understanding of the evaluand. An understanding of the intricacy of human values, however, is not only restricted to the evaluand’s lifeworld but is equally important to the national or international organization in charge of the evaluation. Organizations codify standards and good practices and are constantly challenged by whose values are to be considered. Value-based evaluation (VBE) explores all stakeholders’ values to “value the values” in a particular evaluation. Values are hard to measure but can be rewarding in terms of what they bring about understanding and unpacking the fundamental changes or the fact that “no changes” have occurred at all.

This is more than important when working in contexts of fragility or conflict, or with complex subjects like rural development, gender equality, humanitarian crises, transformational change. However, incorporating these values requires diligent effort and presents challenges due to their subjective nature, unreliability, lack of empirical testing, and inherent complexity. Moreover, values become even more difficult to engage with when applied in fragile, violent and turbulent environments, as in ongoing intra- or intergroup conflicts, interstate conflicts or in post-conflict states. In these environments, a range of development projects are planned and implemented to improve living conditions, for example infrastructure, health, education and peace projects that are evaluated from short-term and long-term perspectives. The challenges for evaluation in these contexts are well addressed in the book titled Evaluation in Contexts of Fragility, Conflict and Violence. Guide from Global Evaluation Practitioners (Hassnain et al., 2021).  

What is value-based evaluation?

For this blog, we have defined VBE as: “an approach based on primarily, but not exclusively, the values of evaluand and the subject of evaluation. Value-based evaluation is more than participation and cultural sensitivity. It is an understanding of deeply rooted values inside the community or an institution” (Aronsson & Hassnain, 2019). Our exploration of VBE is a continuation of our prior interest in the topic (see Aronsson & Hassnain, 2019; Naidoo & Soares, 2017; Uitto & Batra, 2020). 

Why does value-based evaluation matter?

First and foremost, it directly affects people’s safety, making it a matter of utmost importance for any development or humanitarian policy or programme. Secondly, a significant number of human development interventions experience either failure or inefficiency, emphasizing the need for improvement. Thirdly, VBE helps ensure accountability and transparency at all levels, especially by making people and communities change agents, promoting stakeholder engagement and thus ultimately enhancing the overall effectiveness of interventions (Patton, 2008). Fourthly, if evaluation is to stay relevant and cutting-edge, exploration of complex issues such as addressing the preferences and aspirations of the communities must advance, in order to foster more inclusive and sustainable outcomes. Equally important is to pay attention to the values of groups or institutions worked with, as these may directly affect on or be affected by the community values.

Progress has been made; however, too many development projects still fail. An example consists in the well-documented difficulties with resettlement of people because of the construction of large dams. Something is missing in the understanding of these projects that seems to be related to the “soft” values of culture and society. A soft value is something that is not obviously related to the project; it seldom becomes visible or articulated in formal interviews and in questionnaires, but is nevertheless present and influences the project. Intangible cultural heritage and memories can belong to this category, but also sensitive cultural issues such as belief in witchcraft, or hierarchical working relationships and nepotism.     

A soft value can be harder to comprehend, but any such intangible value can, with time, materialize with the full force of symbolism and trigger all kinds of behaviour. It is difficult to foresee (and control?) this undercurrent of values that emerges to the surface of a society in a seemingly erratic way, particularly in fragile and conflict contexts. These soft values’power can be conceptualized as the imagery of the village, with those soft values often, but not automatically, more powerful than the village’s mere physical fabric.  

Unveiling the challenges of value-based evaluation

The study of values is not new. On the contrary, it shows breadth and sophistication going back at least 2,000 years. However, it also reveals inconsistencies, especially in relation to evaluation work. Value is claimed to be at the core of evaluation (Fournier, 2005), but this is also a “misconception” (Patton, 2023). The term “value” is used in a vague way, is difficult to operationalize, and the actions taken based thereupon may be ineffective. This became apparent at the INTEVAL meeting in Rome, hosted by IOE and IFAD from 29 to 31 May 2023, where VBE was discussed with a number of development evaluators from around the world. This blogpost is an outcome of the discussions held at the meeting.

The definition of VBE provided by Aronsson and Hassnain (2019), discussed above, and the discussions at the INTEVAL meeting in Rome emphasize the importance of going beyond localized perspectives, such as informed participation, indigenous knowledge, Western hegemonic knowledge, or feminist knowledge. To achieve a comprehensive understanding of change in various contexts, groups or organizations, it is important to transcend compartmentalized knowledge and embrace a holistic approach. This becomes crucial in the age of artificial intelligence (AI) and its reliance on machine processing and interpretation of information. Placing values at the core of analysis, reporting, dissemination and the utilization of evaluation-generated knowledge is imperative. The integration of human, community and organizational values with AI and machine learning further underscores the need for a nuanced understanding of their relationship. Exploring the ethical dimensions and potential biases inherent in AI algorithms can help ensure that the application of technology aligns with human values, fostering responsible and inclusive decision-making processes at all levels.

About a hundred years ago, cultural relativism was necessary to highlight the local culture’s values. Now, it is time to realize that cultural relativism is not the solution, because it bankrupts evaluation, fragments societies and nation-states and makes comparison impossible, since all values are equally valid. All forms of knowledge need to be scrutinized and contextualized. Tradeoffs are necessary between individual, local community and universal values, in order to reach durable transformational change. It is not ethnocentric to claim that standards based on universal values, and that may have been tested and found relevant, are irrelevant for a particular case. In a VBE, emic and etic values must interact, although emic is not, strictly speaking, the local model per se but an analytical construction.  

We have identified two approaches to VBE so far. First, there is an “evaluation of values” in normative evaluation models as practised by large multilateral organizations, e.g.  IOE/IFAD. This would lead to questions on the hierarchy of values, how to conduct evaluation in complex organizations and maybe a challenge of the value consensus of the organization. Second, there is an “incorporation of values” into the evaluation model that would lead to questions on how to identify values, operate competing values, decide whose values to adhere to and accept the consequences.

This is no easy task. The societal processes that create, renegotiate and transform values have to be understood from a structural, but also a subjective, perspective. Personal narratives constitute elements in a social “drama” making up the daily routines of a society. Decision-making processes take place in a given structure. VBE must be realistic, less idealistic, and sometimes accept what is “good enough”. Any misstep or misinterpretation can be “fatal” in a highly complex fragile environment that demands quality of evidence as well as quality of evaluation capacity.

The way forward for value-based evaluation

This blogpost serves as a catalyst for advancing discussions on VBE within the evaluation community. In order to further the agenda of value-driven evaluation and ensure the significance of values in evaluations, we present below a set of key points. These points aim to prioritize the inclusion of marginalized voices, enhance the democratic importance of evaluation, and foster a comprehensive understanding of whose values truly matter. By embracing these principles, we can collectively work towards a more equitable and impactful evaluation practice at all levels.

• Contextualizing values: Recognizing the complexity of values, evaluators should better understand, and adopt a nuanced approach that acknowledges, the interplay between the individual, local community, and organizational values they are dealing with. Understanding the societal processes that shape and transform values may require analysing both structural and subjective perspectives. When aiming for idealistic outcomes, evaluators must also be realistic and accept that, in contexts of fragility, conflict and violence, evaluations may need to accept being simply “good enough” in order to effectively navigate highly complex and fragile environments.
• Integration of values into the evaluation criteria: To ensure that values are a fundamental part of development and humanitarian evaluations, it is crucial to explicitly incorporate them into the evaluation criteria and frameworks. This may mean revisiting the OECD DAC evaluation criteria, evaluation guidance from both bilateral and multilateral agencies, academia, and civil society organizations. By explicitly recognizing the importance of values, these frameworks can guide evaluators in considering whose voices and values matter in the evaluation process and in harvesting findings and recommendations that are “useful” and “used”.
• Integrating values into evaluation questions: Evaluation questions should explicitly incorporate values. By aligning evaluation questions with the values relevant to the context, evaluation processes can uncover and assess the impact of these values on development outcomes, shedding light on their influence(s) and implications. This integration of values into evaluation questions fosters a more holistic and comprehensive understanding of the relationships between values and development, enabling evaluators to capture the nuanced dynamics and complexities involved.
• Value assessment, because understanding is not enough: Even if the evaluator grasps the complexity of values in a given context, this is not enough. A value assessment must be carried out to achieve amplified learning and powerful implementation. Assessing value is critical for describing and perceiving what is important. There are many things that are easier to assess than value. For example, it is relatively easy to measure cost and hours worked, discover defects found, or count doorknobs and categorize kitchen walls, yet fail to see the house as an image of the community economy (Aronsson, 2002). These other measurements are subordinated to value, which leads to the tension that arises between values.    
• Empowering marginalized voices: A value-driven evaluation approach should strive to bring to the surface the voices of marginalized communities and individuals. This can be achieved by adopting inclusive and participatory methodologies that actively engage with marginalized stakeholders throughout the evaluation process. It is important to create spaces for these voices to be heard, ensuring that their perspectives, needs and aspirations are given due consideration in evaluation findings and recommendations. Promoting a democratic and inclusive approach to evaluation can contribute to equitable and sustainable development outcomes.
• Balancing of values: For decades, informed participation has been the accepted method to incorporate marginalized voices, at least in internationally funded projects with guiding policy documents. The use of a VBE must develop a sound relationship to pitfalls such as excessive reliance on definitions and conceptualization, the use of an indiscriminate mix of empirical data and speculations, the cult of local knowledge, and belief in a flawless science (Aronsson & Gumucio, 2009).
• Understanding the relationship between values and technology: Given the increasing role of AI and machine learning in evaluation, it is essential to critically examine the ethical dimensions and potential biases inherent in these technologies. Evaluators must explore how values interact with AI algorithms, ensuring that technology aligns with human values and does not perpetuate inequities or exclusion. This necessitates ongoing dialogue, research and capacity development to navigate the evolving relationship between human and organizational values and technological advancements in evaluation practice. VBE offers a mediating role between emerging AI technologies and real people on the ground.

By integrating values into evaluation criteria, empowering marginalized voices, understanding the relationship between values and technology, and embracing contextual trade-offs, a value-driven evaluation approach can foster meaningful and inclusive development outcomes. Such an approach acknowledges the importance of human and organizational values, while also recognizing the need for adaptability and responsiveness to diverse sociocultural, political and economic contexts.

Conclusion

We emphasize that evaluation loses its relevance, usefulness and comprehensibility when devoid of human and organizational values. However, it is imperative to study values in their unprocessed state, for example from their raw materials and not from an a priori position, a predetermined standpoint. It is necessary to discuss differences of values as placed in their sociocultural, sociopolitical and socioeconomic context, to reach a legitimized evaluation that is resonant to the perspectives of a community, a national institution and international donors. This entails recognizing the essential preconditions involving governance and accountability receptivity, while also keeping in mind the humanistic aspect of evaluation constantly when venturing into the challenging territory of evaluating in contexts of fragility, including environments affected by natural or manmade crises.

Authors

Indran A. Naidoo, Director, Independent Office of Evaluation of IFAD [here]

Inga-Lill Aronsson, Senior Lecturer/Associate Professor, Uppsala University [here]

Hur Hassnain, Senior Evaluation Advisor (Deputy Team Leader), European Commission, DG INTPA, Evaluation Support Service [here]

References

Aronsson, I-L. & Hassnain, H. (2019). “Value-based evaluations for transformative change”. In: van den Berg, R., Magro, C., Salinas Mulder, S. (eds). Evaluation for Transformational Change. IDEAS: Exeter, UK.

Aronsson, I-L. & Gumucio, JC. (2009). In search for a common ground in the encounter between technical expertise and indigenous knowledge. Paper presented at the World Conference of Humanitarian Studies (WCHS), University of Groningen, February 4 – 8, 2009. Panel: Local Participation and Knowledge Production. Unpublished paper. 

Aronsson, I-L. (2002). Negotiating Involuntary Resettlement. A study of Local Bargaining during the Construction of the Zimapán Dam. Doctoral thesis. Department of Cultural Anthropology and Ethnology, Uppsala University. https://uu.diva-portal.org/smash/get/diva2:161223/FULLTEXT01.pdf

Hassnain, H. & Kelly, L. & Somma, S. (eds) (2021).  Evaluation in Contexts of Fragility, Conflict and Violence. IDEAS: Exeter, UK. 

Fournier, D. (2005) “Evaluation”. In: Mathison, S. (ed.) Encyclopedia of Evaluation. Thousand Oaks, CA: Sage Publication, 139-140.

Naidoo, I. & Soares A. R(2017). Incorporating the Sustainable Development Goals in National Evaluation Capacity Development. In: van den Berg, R.D., Naidoo, I., Tamondong, S.D. (eds). Evaluation for Agenda 2030. IDEAS & UNDP.

Patton, M.Q. (2008). Utilization-Focused Evaluation: The New Century Text. Thousand Oaks, CA: Sage Publications, United States

Patton, M.Q. (2023). Evaluation. 7 common evaluation misconceptions. 1 reconceptualization. 12 January 2023. Video resource. https://www.youtube.com/watch?v=iMMupAh9-JE

Rokeach, M. (1973). The Nature of Human Values. Free Press. New York. USA.

Uitto, J. & Batra G. (eds). (2020). Transformational Change for People and Planet. Sustainable Development Goal Series. Springer. Switzerland. Available at: https://link.springer.com/content/pdf/10.1007%2F978-3-030-78853-7.pdf