CEP to deliver a new project on the evaluation of Biodiversity Net Gain

CEP has been awarded a new project by Natural England and Defra on the design of an evaluation framework for Biodiversity Net Gain in England

CEP, in partnership with BSG Ecology, Geodata Institute, CECAN and Vivid Economics, has been commissioned by Natural England on behalf of Defra to undertake a new project to design an evaluation framework for Biodiversity Net Gain (BNG) in England. The aim of mandatory BNG policy, as set out in the Environment Bill, is to secure a measurable improvement in habitat for biodiversity whilst streamlining the planning process and creating better places for local communities.

The purpose of the evaluation framework is to enable Natural England and Defra to evaluate how BNG is being delivered and understand the impact (environmental, social, and economic) of the BNG policy.

In this project we will conduct a streamlined evidence review to bring together the most up to date relevant evidence on key issues related to BNG. This evidence will feed into the development of the evaluation framework, including setting out the underpinning intervention theory by examining the logic of mandatory BNG and the intervention pathways associated with delivering BNG in practice. Development of the framework will be further supported by work to scope the data requirements and potential sources. The process will be iterative, incorporating a range of stakeholder perspectives from developers, local planning authorities, conservation, amenity and local community NGOs and partnerships as well as central government.

The project started in September 2020 and is expected to run until March 2021.

For more information, please contact Paula Orr (Technical Director, CEP) or Dr Sian Morse-Jones (Principal Consultant, CEP).

Evaluating policy interventions: What role for Theory of Change?

Evaluating policy interventions: What role for Theory of Change?

Blog post by Sian Morse-Jones

What is a Theory of Change (ToC)?  Different terminologies/approaches exist. From an evaluation perspective a ToC commonly articulates how an intervention (e.g. policy, programme, project) is expected to lead to an ultimate goal(s) by showing what needs to happen, in what order and in what way. It establishes the ‘how’ and ‘why’ activities lead to outputs, outcomes and ultimately goals/impacts, explaining the assumptions underpinning this. Usually presented in a diagram or map, a ToC allows big picture thinking, and can help to contextualise where an intervention sits alongside other influences, depicting how external factors may also influence the goal. 

Example Theory of Change diagram for Countryside Stewardship Facilitation Fund (CEP, 2015, Report to Defra)

In evaluation, having a clear understanding of an intervention’s ToC is incredibly useful for a variety of reasons. It can clarify the causal relationships between different activities, outputs, outcomes and goals and highlight the chief assumptions which underpin why these contribute to specific goals. This can provide pertinent information to inform the evaluation framework, questions, criteria, evidence and needs.

A frequent challenge in the evaluation of policy interventions is that impacts/goals are often long term, for example, the biodiversity benefits from habitat improvement or creation may take years to materialise, or as in the case of emergencies such as flooding, may not manifest within the timeframe of the evaluation. The ToC is a powerful tool in such contexts because it provides a theoretical basis for evaluating these in terms of activities, outputs or outcomes which are measurable.

Evaluation of an intervention also provides the opportunity to further test and explore the validity of the ToC, enabling further refinements. Because the ToC tells us that achieving goals rely on specific assumptions, it can be very helpful to explore and consider whether assumptions are correct, why, and in what circumstances? Similarly, if assumptions are not correct, why are they not? What’s not working? In this way, a ToC-based evaluation can help to surface valuable lessons to benefit the design of an intervention, as well as informing policy and practice.

CEP has  much experience in using ToC to evaluate policy interventions – applications include:

• Our work on Our Bright Futures  to evaluate how, and to what extent, a programme and portfolio of projects aimed at young people, has led to progressive change in outcomes for the young people, the environment, their communities and the economy, as well as the long-term influence and legacy.

• In the monitoring and evaluation of Nature Improvement Areas for Defra, and

• In evaluating a project on community engagement on flood risks for Natural Resources Wales.

In CEP’s experience, understanding the theory behind ‘how’ and ‘why’ an intervention will have an impact in the real world is a vital part not just of designing effective interventions, but also in evaluation. Yet it is surprisingly rare for policy interventions to be well articulated in a ToC, often because an explicit policy cycle / clear role for how evaluation can be used in future policy is lacking[1].  Getting the ToC right at the outset may not only result in a stronger intervention, but also a more robust and efficient evaluation process, drawing out key lessons for the future direction of policy and implementation.

Dr Sian Morse-Jones, Senior Consultant, CEP, 22 October 2018

[1] For more details see CEP’s meta-evaluation of 10 years’ of our projects: Learning the Lessons for Evaluating Complexity across the Nexus.

CEP attending CECAN annual conference

CEP attending CECAN's annual conference on policy evaluation in London on July 11th

Dr Bill Sheate will be attending CECAN's annual conference in London on July 11th. The theme of the conference this year is Policy Evaluation for a Complex World - The Challenges that Complexity Poses for Policy - Solutions and Benefits. 

Bill led a meta-evaluation study for CEP - Learning lessons for evaluating complexity across the nexus: a meta-evaluation of CEP projects - published in 2017 as part of our involvement as partners in CECAN.

CEP presenting CECAN fellowship work

CEP presenting work on Flood Community Resilience Pathfinder evaluation at Defra Theme Advisory Groups meeting

Dr Clare Twigger-Ross will be giving a brief presentation on her CECAN fellowship work looking at Qualitative Comparative Analysis in relation to the Flood Community Resilience Pathfinder evaluation at the Defra/Environment Agency R&D Theme Advisory Groups meeting in Birmingham on 26th April.

CEP presenting at the European Environmental Evaluators Network Forum

Dr Clare Twigger-Ross is presenting at the European Environmental Evaluators Network Forum in Edinburgh, 23 -24th November 

The 2017 Forum will ask whether evaluation is evolving in line with the societal and environmental challenges policy is trying to address. Europe, in order to achieve its goal of “living well, within the limits of our planet”, will need to rely inter alia on innovation to make the transition to sustainability. Many forms of innovation will be required to turn Europe into a resource-efficient, green and low-carbon economy.  

Clare is presenting  work carried out by a team at CEP (Dr Bill Sheate, Dr Clare Twigger-Ross, Owen White, Rolands Sadauskis, Paula Orr, Liza Papadopoulou and Ric Eales):  "Learning Lessons from evaluations across the nexus: a meta-evaluation", which examined CEP evaluations in the context of complexity and impact, developing insights and key questions for future evaluations.   This is being presented as part of a session from CECAN on Innovations in instrument evaluation - integrating complexity into Environmental policy evaluation: Insights from CECAN on Thursday afternoon.   The project report and summary can be found here.

CEP at CECAN meeting

CEP's Dr Clare TwiGger-Ross at CECAN team meeting

Dr Clare Twigger-Ross is attending the internal CECAN (Centre for the Evaluation of Complexity across the Nexus) team meeting at Barnett Hill, near Guildford on Friday 13th October .  This meeting will cover CECAN case studies,  updates from CECAN fellows, and discuss the impact of CECAN in terms of outputs, capacity building in both practitioners and policy makers, and policy relevance.

For further information about CEP's involvement in CECAN see our news items here.

Further information about CECAN can be found at  www.cecan.ac.uk

CEP at CECAN Workshop on increasing the impact of policy evaluation

CEP at CECAN Workshop on increasing the impact of policy evaluation

CEP participated in a workshop organised by CECAN to explore how evaluation practice can become more impactful in the future. The workshop 'Increasing Impact of Policy Evaluation in Complex Settings – What Works?' took place on 5 September 2017 in London. It sought to:

1. share recent experience on successful evaluation impact in complex nexus policy areas
2. share expertise in maximising the impact of evaluation in these policy settings
3. generate practical recommendations for policymakers, analysts and evaluators
4. generate ideas for focussing future R&D in this area
5. produce a policy note on how evaluation practice can become more impactful in future

CEP's  Dr Clare Twigger-Ross discussed the impact of CEP's evaluation of Defra's Flood Resilience Community Pathfinder.

Further details of the workshop can be found here.

CEP'S SEMINAR ON PRACTICAL POLICY EVALUATION AVAILABLE ONLINE

CEP'S SEMINAR ON LEARNING LESSONS FROM PRACTICAL POLICY EVALUATION AVAILABLE ONLINE

CEP's Dr Clare Twigger-Ross, Owen White and Dr Bill Sheate delivered a seminar for the Centre for the Evaluation of Complexity Across the Nexus (CECAN) on Learning lessons from practical policy evaluation. The seminar reflected on the findings of a meta-evaluation study of 23 of CEP's evaluation projects, exploring lessons around the evaluation of complexity, the role of methods and the nature of evaluation impact (full report available here). 

To listen to the discussion on the key factors that affect evaluation progress and gain insights on managing complexity and navigating an evaluation through dynamic policy landscapes, click here.

CEP to deliver CECAN seminar on practical policy evaluation

CEP to deliver CECAN seminar on learning lessons from practical policy evaluation

CEP will be delivering a seminar for the Centre for the Evaluation of Complexity Across the Nexus (CECAN) on Learning lessons from practical policy evaluation: reflecting on a meta-evaluation of UK/EU policy and practice evaluations across the nexus.

The seminar will take place on 14 June 2017, in London, and will reflect on the findings of a meta-evaluation study of 23 of CEP's evaluation projects, exploring lessons around the evaluation of complexity, the role of methods and the nature of evaluation impact. This research brought out the key factors that affect evaluation progress and provided insights on how to navigate an evaluation through dynamic policy landscapes. The full report is available here.

In this seminar CEP's Dr Clare Twigger-Ross and Owen White will facilitate a discussion around the key findings from this research and approaches for managing complexity in evaluations. 

For more information and to register for this seminar visit CECAN's website.

Learning lessons for evaluating complexity across the nexus

CECAN (the Centre for Evaluation of Complexity across the Nexus, based at the University of Surrey, has published the final report of CEP's meta-evaluation study of 23 selected evaluation projects we have undertaken relating to Nexus issues – water, environment, food, energy - over the last 10 years.  

The full report  - Learning lessons for evaluating complexity across the nexus: a meta-evaluation of CEP projects - is available here.

We evaluated the evaluation approaches and findings from a range of case studies – national and EU policy level down to programme level policy interventions and other initiatives – and sought to address three aims:
1.    To learn the lessons from past policy evaluations; 
2.    To understand the factors that support or inhibit (barriers or enablers to) successful evaluations: and
3.    To explore the value of different types of approaches and methods used for evaluating complexity

An important finding was the extent to which the contexts for evaluations at the EU level and UK levels differ: a very strong policy cycle exists for EU evaluations, which creates a more rigid framework for monitoring and evaluation, compared to the much greater degree of policy flux in the UK, and the resulting need for greater flexibility in the way in which evaluations are undertaken, and hence the greater utility of qualitative data collection and analysis methods.

Consequently the use or influence of evaluations in policy making differs considerably – there is much more instrumental (direct) use of EU evaluations compared to more conceptual or process (indirect) use of UK evaluations.

For further information contact Dr Bill Sheate or Dr Clare Twigger-Ross 

The key findings from the study are summarised in the infographic below:

CEP's Dr Clare Twigger-Ross awarded CECAN Fellowship

Photo credit: 'jigsaw puzzle pieces' by Electric-Eye on Flickr (CC BY 2.0)

CEP's Dr Clare Twigger-Ross awarded CECAN Fellowship

CEP's Dr Clare Twigger-Ross has been awarded a Fellowship by the Centre for the Evaluation of Complexity Across the Nexus (CECAN) Investigating the value of qualitative comparative analysis in relation to the Defra Flood Resilience Community Pathfinder (FRCP) evaluation. Building on the Defra FRCP evaluation undertaken by CEP (2013 - 2015), this study aims to identify key factors that predict improved community resilience.

The key evaluation challenge for this study will be the attribution of causality in the context of community resilience to flooding. A key methodological development will be testing qualitative comparative analysis (QCA) in a novel (flooding) complex evaluation situation to examine its feasibility for similar situations, and to assess the aspects of QCA of most use to this type of evaluation.

This is expected to raise key complexity issues in evaluation, which have been recently explored in CEP's project for CECAN Learning lessons for evaluating complexity at the nexus: a meta-evaluation of CEP projects. The findings of this project will become available on the CECAN website in the coming months.

CEP undertaking a meta-evaluation of its past policy evaluations

 

Photo credit: 'puzzle' by Kevin Dooley on Flickr (CC BY-SA 2.0)

CEP undertaking a ‘meta-evaluation’ of its past policy evaluations for CECAN

As part of CEP’s role as a partner in CECAN (Centre for the Evaluation of Complexity Across the Nexus), CEP is carrying out a meta-evaluation of a sample of the evaluation projects it has undertaken over the last 10 years. The review will seek to learn the lessons from past policy evaluations, and the value of different types of approaches and methods used for evaluating complexity.

This extensive meta-evaluation of projects will support CECAN’s initial scoping stage and provide critical insights in understanding complexity and developing new ways to measure the effectiveness of policies across the ‘nexus’.

CEP’s Dr Clare Twigger-Ross and Dr Bill Sheate are joint project leads.

CEP at CECAN workshop on understanding complexity

CEP at CECAN workshop on understanding complexity

CEP's Dr Clare Twigger-Ross and Dr Bill Sheate will participate in a 1 day workshop on 9th June hosted by the Centre for the Evaluation of Complexity Across the Nexus (CECAN) to tackle the question of 'What is complexity?'.

 Discussions will include:

• What are our different understandings of complexity?
• Examples of complexity in policy evaluation and design, and Nexus systems
• What factors in policy processes and Nexus systems make complexity difficult to handle?
• Sharing innovative ideas and approaches
• What tools can we bring/matching of tools to complex problems
• As a policy maker, how do I know I am dealing with a complex problem?

The workshop will focus on some of the real-world policy case studies put forward by CECAN's co-funders and provide an opportunity to plan a wider public 2 day residential workshop in September 2016.

The event is by invitation only.

CEP PART OF THE PROJECT TEAM EVALUATING COMPLEXITY ACROSS THE NEXUS

CEP part of project team for the ESRC Centre for the Evaluation of Complexity across the Nexus

CEP is part of the  consortium of leading UK bodies who have initiated a new national research hub to develop new ways to measure the effectiveness of domestic policies on energy, water, environment and food (the ‘nexus’), and how they affect wider society.

The focus of the Centre for the Evaluation of Complexity Across the Nexus (CECAN) is to pioneer, test and promote evaluation approaches and methods across the nexus where complexity (e.g. of the issues and governance) presents a challenge to policy interventions, and so contribute to more effective policy-making.

CEP's Dr Clare Twigger-Ross and Dr Bill Sheate are members of the project team.

For more information see the ESRC press release.