New Magenta Book 2020 Supplementary Guide on Handling Uncertainty in Policy Evaluation
Blog Post by Paula Or
The Centre for the Evaluation of Complexity at the Nexus (CECAN)’s Guide on Handling Uncertainty in Policy Evaluation which was published by HM Treasury at the beginning of April, throws a virtual lifeline to those buffeted by the current storms of uncertainty who are still seeking to make meaningful evaluations of policy interventions.
CECAN was commissioned to write the Supplementary Guide for the Magenta Book - the cross-government guidance on evaluation. The Guide is the product of three years’ research and development of evaluation methods by CECAN. CEP was part of the first CECAN consortium led by University of Surrey between 2016-2019.
Understanding how to evaluate policy interventions characterised by complexity and uncertainty is crucial to CEP’s work. Our experience of policy evaluation spans forward-looking / prospective or ex ante evaluation; process, outcome, impact and economic evaluations; and reflective or ex post evaluations. CECAN describes complexity as ‘made up of many diverse components that interact with each other in nonlinear ways and can adapt’ (Source: CECAN, 2020). Most of the areas CEP evaluates are complex; many of our evaluations explore the interactions between natural and social systems.
The new Supplementary Guide provides greater understanding of complexity and its challenges as well as providing practical tools for those who commission these evaluations and the practitioners who undertake them.
Policy is developed with the intention of managing systems. The more complex these systems, the more difficult they are to manage. For example, path dependency is a property of complex systems which means that the way the system develops in the future depends on how it got to where it is as well as on its current state. In the natural world, organisms mutate and adapt from what they were, they cannot radically change. In social systems, we are getting better at recognising that important changes in behaviour, for example, are rarely produced by single actions (such as making people aware of the need to change or providing an economic incentive to change) but require a whole sequence of processes or changes across overlapping systems (material, social and individual).
CECAN’s Supplementary Guide usefully describes eleven properties of complex systems, providing diagrams of each, to help create a common language or reference points for people from different disciplines.
Evaluation is crucial to help understand and navigate complexity. Evaluations of interventions involving complex systems and uncertainty benefit from building in learning throughout the process - and recognising that learning may bring with it a need to adapt or tweak the evaluation to ensure that it is still focusing on the right questions and collecting relevant information. CEP’s experience of using learning as a tool for going beyond the question ‘What works?’ to understand ‘What has changed?’ and explore the reasons for unexpected outcomes has generated unexpected insights. Examples are the evaluations of the Catchment-Based Approach and the Flood Resilience Community Pathfinders, both carried out for Defra.
Evaluation can also help to bring stakeholders into the process of planning and managing complexity in interventions. Stakeholders can help to make sense of how the existing situation came about and relationships within the system, for example. This can suggest ways that change might happen in the future. This gives access to deeper understanding as well as building a sense of agency and ownership.
CECAN’s Policy and Practice Note on Learning lessons from complex evaluations across the nexus, led by CEP, found that time is a key element in complexity and must be taken into account in designing evaluations. This theme is taken up in the Supplementary Guide:
it is difficult to predict at the start of an intervention what change will happen or how long it will take for evidence of change to emerge
change may continue longer than expected and usually goes on well beyond the end of a project or intervention
it can take time for ‘complexity’ features to become apparent in a policy intervention.
This has important implications for the design of evaluations. Both those who commission and those who design and carry out evaluations need to adopt a different ‘mindset’, taking an adaptive management approach to interventions, with ‘evaluative practice’ happening alongside and evolving with it.
In the current uncharted waters, the Supplementary Guide does not offer a path to calm seas - quite the opposite. It asks us all to be more agile and flexible to change. What it does offer is an immensely valuable set of insights and tools to help make sense of the signs and processes along the way in order to navigate uncertainty.