Evaluating policy interventions: What role for Theory of Change?
Blog post by Sian Morse-Jones
What is a Theory of Change (ToC)? Different terminologies/approaches exist. From an evaluation perspective a ToC commonly articulates how an intervention (e.g. policy, programme, project) is expected to lead to an ultimate goal(s) by showing what needs to happen, in what order and in what way. It establishes the ‘how’ and ‘why’ activities lead to outputs, outcomes and ultimately goals/impacts, explaining the assumptions underpinning this. Usually presented in a diagram or map, a ToC allows big picture thinking, and can help to contextualise where an intervention sits alongside other influences, depicting how external factors may also influence the goal.
Example Theory of Change diagram for Countryside Stewardship Facilitation Fund (CEP, 2015, Report to Defra)
In evaluation, having a clear understanding of an intervention’s ToC is incredibly useful for a variety of reasons. It can clarify the causal relationships between different activities, outputs, outcomes and goals and highlight the chief assumptions which underpin why these contribute to specific goals. This can provide pertinent information to inform the evaluation framework, questions, criteria, evidence and needs.
A frequent challenge in the evaluation of policy interventions is that impacts/goals are often long term, for example, the biodiversity benefits from habitat improvement or creation may take years to materialise, or as in the case of emergencies such as flooding, may not manifest within the timeframe of the evaluation. The ToC is a powerful tool in such contexts because it provides a theoretical basis for evaluating these in terms of activities, outputs or outcomes which are measurable.
Evaluation of an intervention also provides the opportunity to further test and explore the validity of the ToC, enabling further refinements. Because the ToC tells us that achieving goals rely on specific assumptions, it can be very helpful to explore and consider whether assumptions are correct, why, and in what circumstances? Similarly, if assumptions are not correct, why are they not? What’s not working? In this way, a ToC-based evaluation can help to surface valuable lessons to benefit the design of an intervention, as well as informing policy and practice.
CEP has much experience in using ToC to evaluate policy interventions – applications include:
Our work on Our Bright Futures to evaluate how, and to what extent, a programme and portfolio of projects aimed at young people, has led to progressive change in outcomes for the young people, the environment, their communities and the economy, as well as the long-term influence and legacy.
In the monitoring and evaluation of Nature Improvement Areas for Defra, and
In evaluating a project on community engagement on flood risks for Natural Resources Wales.
In CEP’s experience, understanding the theory behind ‘how’ and ‘why’ an intervention will have an impact in the real world is a vital part not just of designing effective interventions, but also in evaluation. Yet it is surprisingly rare for policy interventions to be well articulated in a ToC, often because an explicit policy cycle / clear role for how evaluation can be used in future policy is lacking[1]. Getting the ToC right at the outset may not only result in a stronger intervention, but also a more robust and efficient evaluation process, drawing out key lessons for the future direction of policy and implementation.
Dr Sian Morse-Jones, Senior Consultant, CEP, 22 October 2018
[1] For more details see CEP’s meta-evaluation of 10 years’ of our projects: Learning the Lessons for Evaluating Complexity across the Nexus.