policy-making

Evaluating policy interventions: What role for Theory of Change?

Evaluating policy interventions: What role for Theory of Change?

Blog post by Sian Morse-Jones

What is a Theory of Change (ToC)?  Different terminologies/approaches exist. From an evaluation perspective a ToC commonly articulates how an intervention (e.g. policy, programme, project) is expected to lead to an ultimate goal(s) by showing what needs to happen, in what order and in what way. It establishes the ‘how’ and ‘why’ activities lead to outputs, outcomes and ultimately goals/impacts, explaining the assumptions underpinning this. Usually presented in a diagram or map, a ToC allows big picture thinking, and can help to contextualise where an intervention sits alongside other influences, depicting how external factors may also influence the goal. 

Example Theory of Change diagram for Countryside Stewardship Facilitation Fund (CEP, 2015, Report to Defra)

Example Theory of Change diagram for Countryside Stewardship Facilitation Fund (CEP, 2015, Report to Defra)

In evaluation, having a clear understanding of an intervention’s ToC is incredibly useful for a variety of reasons. It can clarify the causal relationships between different activities, outputs, outcomes and goals and highlight the chief assumptions which underpin why these contribute to specific goals. This can provide pertinent information to inform the evaluation framework, questions, criteria, evidence and needs.

A frequent challenge in the evaluation of policy interventions is that impacts/goals are often long term, for example, the biodiversity benefits from habitat improvement or creation may take years to materialise, or as in the case of emergencies such as flooding, may not manifest within the timeframe of the evaluation. The ToC is a powerful tool in such contexts because it provides a theoretical basis for evaluating these in terms of activities, outputs or outcomes which are measurable.

20150906_174810.jpg

Evaluation of an intervention also provides the opportunity to further test and explore the validity of the ToC, enabling further refinements. Because the ToC tells us that achieving goals rely on specific assumptions, it can be very helpful to explore and consider whether assumptions are correct, why, and in what circumstances? Similarly, if assumptions are not correct, why are they not? What’s not working? In this way, a ToC-based evaluation can help to surface valuable lessons to benefit the design of an intervention, as well as informing policy and practice.

CEP has  much experience in using ToC to evaluate policy interventions – applications include:

  • Our work on Our Bright Futures  to evaluate how, and to what extent, a programme and portfolio of projects aimed at young people, has led to progressive change in outcomes for the young people, the environment, their communities and the economy, as well as the long-term influence and legacy.

  • In the monitoring and evaluation of Nature Improvement Areas for Defra, and

  • In evaluating a project on community engagement on flood risks for Natural Resources Wales.

shutterstock_120960130 Flooding York.jpg

In CEP’s experience, understanding the theory behind ‘how’ and ‘why’ an intervention will have an impact in the real world is a vital part not just of designing effective interventions, but also in evaluation. Yet it is surprisingly rare for policy interventions to be well articulated in a ToC, often because an explicit policy cycle / clear role for how evaluation can be used in future policy is lacking[1].  Getting the ToC right at the outset may not only result in a stronger intervention, but also a more robust and efficient evaluation process, drawing out key lessons for the future direction of policy and implementation.

Dr Sian Morse-Jones, Senior Consultant, CEP, 22 October 2018


[1] For more details see CEP’s meta-evaluation of 10 years’ of our projects: Learning the Lessons for Evaluating Complexity across the Nexus.

 





CEP at CECAN meeting

CEP's Dr Clare TwiGger-Ross at CECAN team meeting

Dr Clare Twigger-Ross is attending the internal CECAN (Centre for the Evaluation of Complexity across the Nexus) team meeting at Barnett Hill, near Guildford on Friday 13th October .  This meeting will cover CECAN case studies,  updates from CECAN fellows, and discuss the impact of CECAN in terms of outputs, capacity building in both practitioners and policy makers, and policy relevance.

For further information about CEP's involvement in CECAN see our news items here.

Further information about CECAN can be found at  www.cecan.ac.uk

CEP'S SEMINAR ON PRACTICAL POLICY EVALUATION AVAILABLE ONLINE

CEP'S SEMINAR ON LEARNING LESSONS FROM PRACTICAL POLICY EVALUATION AVAILABLE ONLINE

CEP's Dr Clare Twigger-Ross, Owen White and Dr Bill Sheate delivered a seminar for the Centre for the Evaluation of Complexity Across the Nexus (CECAN) on Learning lessons from practical policy evaluationThe seminar reflected on the findings of a meta-evaluation study of 23 of CEP's evaluation projects, exploring lessons around the evaluation of complexity, the role of methods and the nature of evaluation impact (full report available here). 

To listen to the discussion on the key factors that affect evaluation progress and gain insights on managing complexity and navigating an evaluation through dynamic policy landscapes, click here.

Learning lessons for evaluating complexity across the nexus

CECAN (the Centre for Evaluation of Complexity across the Nexus, based at the University of Surrey, has published the final report of CEP's meta-evaluation study of 23 selected evaluation projects we have undertaken relating to Nexus issues – water, environment, food, energy - over the last 10 years.  

The full report  - Learning lessons for evaluating complexity across the nexus: a meta-evaluation of CEP projects - is available here.

We evaluated the evaluation approaches and findings from a range of case studies – national and EU policy level down to programme level policy interventions and other initiatives – and sought to address three aims:
1.    To learn the lessons from past policy evaluations; 
2.    To understand the factors that support or inhibit (barriers or enablers to) successful evaluations: and
3.    To explore the value of different types of approaches and methods used for evaluating complexity

An important finding was the extent to which the contexts for evaluations at the EU level and UK levels differ: a very strong policy cycle exists for EU evaluations, which creates a more rigid framework for monitoring and evaluation, compared to the much greater degree of policy flux in the UK, and the resulting need for greater flexibility in the way in which evaluations are undertaken, and hence the greater utility of qualitative data collection and analysis methods.

Consequently the use or influence of evaluations in policy making differs considerably – there is much more instrumental (direct) use of EU evaluations compared to more conceptual or process (indirect) use of UK evaluations.

For further information contact Dr Bill Sheate or Dr Clare Twigger-Ross 

The key findings from the study are summarised in the infographic below:

CEP meta-evaluation Summary Infographic Jan 2017fin.jpg

CEP's Global MegaTrends report published by the EEA

Photo by: 'Modern Art Sunset' in Thessaloniki by Nubrig on Flickr (CC BY-SA 2.0)

CEP's report on THE DEVELOPMENT OF A METHODOLOGICAL TOOLKIT FOR UNDERSTANDING THE IMPLICATIONS OF GLOBAL MEGATRENDS PUBLISHED BY THE EUROPEAN ENVIRONMENT AGENCY

The European Environment Agency published CEP's report Mapping Europe's environmental future: understanding the impacts of global megatrends at the national level.

This report is the result of CEP's work in developing and testing a methodology to help European member states explore and understand the implications of global megatrends for the environment and for policy at the national level.

The report sets out the logic for identifying the implications of global megatrends at the national, regional or European level, and aims to provide inspiration to EEA members and cooperating countries to undertake their own national studies. It describes the context and the reasons why understanding global trends is important, and sets out a suggested methodology for doing so.

This project was funded by the Swiss Federal Office for the Environment (FOEN) and was delivered under the aegis of the European Environment Agency (EEA) EIONET FLIS.

CEP at annual meeting of the European network for environmental foresight experts

CEP at annual meeting of the European network for environmental foresight experts

CEP’s Owen White has been invited as an external expert to participate in the annual meeting of the EIONET (European Environment Information and Observation Network) forward-looking information and services (FLIS), hosted by the European Environment Agency (EEA) in Copenhagen, Denmark 20-21 September 2016. 

Following on from the FLIS expert workshop Owen presented at in March 2016, Owen will be presenting the methodology CEP has developed related to downscaling the implications of global megatrends at the national or regional scale, and facilitating discussion relating to the piloting of the methodology by six case study EU countries.  The method is intended to enable EU member states or regions to identify and prioritise potential implications of global megatrends, link these with national environmental information and indicators, and help identify emerging environmental policy needs. 

CEP has been contracted to develop this methodology by the Swiss Federal Office for the Environment (FOEN) as part of an ongoing project, and under the aegis of the EEA network of environmental foresight experts.  

CEP undertaking a meta-evaluation of its past policy evaluations

 

Photo credit: 'puzzle' by Kevin Dooley on Flickr (CC BY-SA 2.0)

CEP undertaking a ‘meta-evaluation’ of its past policy evaluations for CECAN

As part of CEP’s role as a partner in CECAN (Centre for the Evaluation of Complexity Across the Nexus), CEP is carrying out a meta-evaluation of a sample of the evaluation projects it has undertaken over the last 10 years. The review will seek to learn the lessons from past policy evaluations, and the value of different types of approaches and methods used for evaluating complexity.

This extensive meta-evaluation of projects will support CECAN’s initial scoping stage and provide critical insights in understanding complexity and developing new ways to measure the effectiveness of policies across the ‘nexus’.

CEP’s Dr Clare Twigger-Ross and Dr Bill Sheate are joint project leads.

EU publishes SEA Directive Effectiveness study

CEP a contributing author TO European Commission Study on the effectiveness of the SEA Directive (Directive 2001/42/EC)

An important review study of the SEA Directive, to which CEP contributed, was published recently by the European Union.  The objective of the Study concerning the preparation of the report on the application and effectiveness of the SEA Directive (Directive 2001/42/EC) was to provide the Commission with information on Member States’ progress and challenges experienced in the application of the Council Directive 2001/42/EC (‘SEA Directive’) on the assessment of the effects of certain plans and programmes on the environment for the period 2007-2014.

The study analysed the practical implementation of the SEA Directive across all 28 Member States and then evaluated the performance of the legislation to understand how environmental considerations have been integrated into planning processes and the extent to which the SEA Directive contributes to achieving better and more coherent planning.  The study found that the SEA Directive has brought about some significant benefits to strategic planning, while also making recommendations that could further improve its efficiency, effectiveness and coherence into the future.

Dr Bill Sheate and Ric Eales from CEP contributed to the study as expert advisors, in association with Milieu Ltd, Brussels.

CEP delivered seminar at James Hutton Institute

Photo credit: 'land lines' by apalca on Flickr (CC BY-SA 2.0)

CEP's Dr Peter Phillips delivered a seminar on land use policy evaluation at JHI

On 28th July, CEP's Dr Peter Phillips gave an invited seminar to members of the James Hutton Institute Social, Economic and Geographical Sciences (SEGS) Research Group. His seminar was entitled Evaluating policies for sustainable land use - methods and case studies and built on CEP's land use and natural environment policy evaluation and research expertise and experience in Scotland and elsewhere. 

CEP at CECAN workshop on understanding complexity

CEP at CECAN workshop on understanding complexity

CEP's Dr Clare Twigger-Ross and Dr Bill Sheate will participate in a 1 day workshop on 9th June hosted by the Centre for the Evaluation of Complexity Across the Nexus (CECAN) to tackle the question of 'What is complexity?'.

 Discussions will include:

  • What are our different understandings of complexity?
  • Examples of complexity in policy evaluation and design, and Nexus systems
  • What factors in policy processes and Nexus systems make complexity difficult to handle?
  • Sharing innovative ideas and approaches
  • What tools can we bring/matching of tools to complex problems
  • As a policy maker, how do I know I am dealing with a complex problem?

The workshop will focus on some of the real-world policy case studies put forward by CECAN's co-funders and provide an opportunity to plan a wider public 2 day residential workshop in September 2016.

The event is by invitation only.

Dr Clare Twigger-Ross at Winter Flood Project findings launch event

Image 'The Somerset Levels' by Nick (CC BY 2.0)

Dr Clare Twigger-Ross to participate in panel discussion on floods, adaptation and community

Dr Clare Twigger-Ross will take part in a panel discussion in an event launching the findings for the Winter Floods Project, undertaken at the Exeter University.

The Winter Floods Project examined how perceptions of the problems and solutions evolved during the year following the floods that occurred over the winter of 2013/14, in order to better understand how longer term policy responses occurs at local and national scales.  The project also investigated individual and community resilience after flood events to identify how flood events impact well-being and quality of life.

The event will present the project’s key findings, followed by a panel discussion on the theme of 'floods, adaptation, and community' with academic and policy contributors. 

The event will take place on 8th June 2016 at the Royal Geographical Society, London between 13:30 and 17:30.

If you would like to attend please use this link to register for a place.   

 

CEP's Dr Clare Twigger-Ross at Evaluation of Complexity Workshop

Image: 'Maze' by Christina Gallivan on Flickr (CC BY 2.0)

CEP’s Dr Clare Twigger-Ross at Evaluation of Complexity workshop

CEP is part of the  consortium of leading UK bodies who have initiated at new national research hub which  will be developing new ways to measure the effectiveness of domestic policies on energy, water, environment and food (the ‘nexus’), and how they affect wider society.

The focus of the Centre for the Evaluation of Complexity Across the Nexus (CECAN) will be to pioneer, test and promote evaluation approaches and methods across the energy, environment and food nexus where complexity presents a challenge to policy interventions, and so contribute to more effective policy-making.

Based at the University of Surrey and launching on 1 March 2016 - prior to a launch event in the summer - CECAN has been backed by £2.45 million of funding provided by the Economic and Social Research Council (ESRC) and the Natural Environment Research Council (NERC) in collaboration with the Department for Environment, Food and Rural Affairs (Defra); the Department of Energy and Climate Change (DECC); the Environment Agency (EA); and the Food Standards Agency (FSA).

Dr Clare Twigger-Ross from CEP will be attending the inception workshop this week Thursday 10 – Friday 11th March.