How to use evidence to identify, learn from, and transfer policy success

One of our project’s aims is deceptively simple: generate evidence to identify how European Union countries try to reduce territorial inequalities, see who is the most successful, and recommend the transfer of that success to other countries.

However, life is not so simple. If it were, we would know for sure what ‘territorial inequalities’ are, what causes them, what governments are willing to do to reduce them, and if they will succeed if they really try.

Potential research problems include an inability to identify:

  • The policies designed explicitly to reduce inequalities. Instead, we piece together many intentions, actions, instruments, and outputs, in many levels and types of government, and call it ‘policy’.
  • The link between ‘policy’ and policy outcomes, because many factors interact to produce those outcomes.
  • Success. Even if we could solve methodological problems, to separate cause and effect in complex systems, we face a political problem about choosing measures to evaluate and report success.
  • Good ways to transfer successful policies. A policy is not like a cake, in which you can produce a great product and give out the recipe. If it were, you could assume that we all have the same aims (we all want cake), starting point (the same kitchen equipment), language to describe the task (use a lot of sugar and cocoa), and choice from clearly define options (chocolate or carrot). However, in policy, governments describe and seek to solve deceptively similar-looking problems in very different ways. Further, if they look to other governments for lessons, those insights have to be relevant to their context. They ‘transfer’ some policies while maintaining their own policies, and the mix is not always coherent. Indeed, a key finding from our previous work is that governments simultaneously pursue policies to reduce inequalities and undermine their inequality-reducing policies.

Academics like me spend their time highlighting these problems, such as in this table explaining why such processes are complex and not ‘evidence-based’ (insights), identifying all the things that will go wrong if you think policymaking and policy transfer can ever be straightforward (issues), and highlighting some ways to respond when presenting evidence (responses).

Insights Issues Responses
‘Evidence based policymaking’ is not a realistic aim Policymakers have a broad view about what counts as good, policy-relevant evidence Produce rich descriptions of problems and solutions based on many forms of knowledge
Policymakers have to ignore almost all evidence Adapt to the cognitive shortcuts of policymakers: minimise the cognitive load of information, frame evidence to help interpret a problem, and reduce uncertainty about the likely effect of solutions.
Policy evaluation has an indirect impact on choice
The policymaking environment:

respond to many actors, institutions, ideas, networks, socioeconomic factors and events

There are many policymakers and influencers spread across government Identify the key venues for authoritative choices.
Each venue has its own ‘institutions’; the formal and informal rules of policymaking Learn the written/ unwritten rules of each venue in which you engage
Each venue is guided by a fundamental set of ideas to determine the nature of problems and feasibility of solutions Learn the language that actors use to frame problems and consider solutions
Each venue has its own relationships between policy makers and influencers Build trust and form alliances within networks
Policymaker attention is driven by changes in socioeconomic factors and events Present solutions during periods of high attention to problems

Policymakers have to make decisions in the face of ambiguity (what problem are they trying to solve?), uncertainty (evidence will help, but always be limited), and limited time. They engage in policy learning and may transfer some policies, but in a time-pressed, complex, and political environment in which (a) they need to adapt new policies to their context, and (b) researchers need to adapt to the ways in which they learn.

Insights Issues Responses
Policy learning Individuals combine cognition and emotion to gain knowledge, and cooperate or compete to prioritise that knowledge in policy Ask: who learns, what do they learn, how, and what is the likely impact on policy change?
Learning is individual and collective, via organisations, coalitions, systems and environments that constrain action Learn the rules of collective action, social dynamics, and external/ political factors key to learning
Categories of learning include: epistemic, reflective, bargaining, hierarchical. Learn how experts relate to: open dialogue, politically salient debates, and actors in authority.
Policy transfer Levels of transfer vary from duplication to broad inspiration Tell a precise story of success: does it relate to an idea or programme?
Transfer ranges from voluntary to coercive Identify what drives policymakers and influences their learning
Transfer can be uninformed, incomplete, or inappropriate Analyse what worked, why, and under what conditions. Ask if those conditions could be replicated.
Policy success incorporates electoral, process, and long term societal outcomes Avoid overly-technical analyses of success. Other conditions include political circumstances and governance arrangements.

 

Our first report on learning and transfer

In that context, our first conceptual report describes practical ways to use academic insights to encourage policy learning across multi-level systems. Our overall aim is to identify, analyse, learn from, and help transfer policies that have reduced territorial inequalities.

Our first objective is to research and share lessons from the governments who project policy success in reducing inequalities in areas such as education attainment, drug-related punishment, and income.

We then describe the policy theories that prompt us to manage our expectations about learning. They explain the relationship between (a) politics, (b) complex policymaking systems, and (c) the lack of ‘evidence-based’ policymaking, learning, and transfer.

It is not feasible to propose a simple evidence-based model for learning, or seek to transfer policy solutions from one region to another without considering their political and policymaking contexts.

Therefore, our second objective is to use our knowledge of policy processes to help produce pragmatic and innovative ways to foster the systematic use of evidence in policy transfer.

Our third objective is to make sure that we have sufficient knowledge of our audience to make an impact with this study. We will:

(a) Produce evidence on the technical feasibility of policy solutions, and

(b) Investigate the ways in which potential importing governments determine their political feasibility.

We will use an iterative process, to identify and generate initial attention for success stories, identify how the case study government defined the policy problem, work with actors in other systems to understand how they would interpret and use the evidence, and use this knowledge to inform our research.

The full conceptual report is available here: https://paulcairney.wordpress.com/2017/11/17/imajine/

 

Paul Cairney