Let’s explore how policy evaluation is used in government.
Evaluation is defined as an appraisal, is backward-looking and asks questions like ‘Did it work?’, whereas policymaking is forward-looking and asks questions like ‘Will this work?’
So, where does evaluation fit into the policy process and how does it all work in reality? How can you evaluate the effectiveness of a policy?
How evaluation works in policymaking
When the government identifies a policy problem, they need to come up with a potential solution, implement and evaluate it. Then, they drop, adapt or scale up the policy based on the evaluation results, as you can see in the ROAMEF Policy Cycle diagram from The UK Magenta Book below:
The ROAMEF Policy Cycle
But if you’ve had any experience working in government, you’ll know that it doesn’t always work like this in practice in the real world.
Rigorous evaluations are rarely done and even when they are, the findings are often not acted upon. Many decisions have to be taken without necessary evidence and it sometimes feels like all the steps of the policy cycle are happening at the same time, without a clear implementation process.
To be a successful public sector policymaker today requires understanding the ideal process as well as the reality in which most governments operate.
Why is policy evaluation important?
It’s also important to talk about the two different uses for evidence within government – accountability and learning. Evidence is useful for both and they are equally important but there are often tensions between the two.
Accountability is about proving something to others. When we speak about holding government accountable for delivering, what we’re asking the government to do is to produce evidence that it has done what it was supposed to. In the context of impact assessment and evaluation, accountability can be about proving to others that a programme was effective or that it works the way its theory of change expects it to.
Learning, on the other hand, is about improving. Here, the consumers of the evidence are often the same ones who are creating it because they want that information to better their policy. The key audience is internal, not external, to those who are responsible for designing and implementing the policy.
Accountability and learning are both important and the same evidence can serve both purposes. For example, an impact evaluation can prove the effectiveness of a programme to outsiders but also help insiders learn ways to improve. There might also be times when there are questions that the insiders are interested in that the outsiders aren’t, or vice versa, and sometimes they even directly conflict.
If evidence is going to be used for accountability purposes, it can make people less open and less willing to learn from the evidence as it means admitting they made mistakes.
Read an example of how impact evaluation works alongside accountability and learning from the proposal in the Center for Global Development report When Will We Ever Learn? Improving Lives Through Impact Evaluation.
Learn to apply policy evaluation in practice
In the Evidence in Public Policy short course, we explore how things should workwhen approaching policy evaluation, both in terms of the technical details of evaluation as well as ways to use evidence. Since it’s worth knowing what an ideal process would look like, we’ll also talk about ways in which reality often makes this difficult or even impossible and what you can do to improve your evidence use in an imperfect world.
Throughout the course, you will be encouraged to keep in mind the dual purposes of evaluation as you study examples of evidence. These dual purposes will also be important when designing your very own impact evaluation on the final module – the course project. You will need to think about how the evidence you create is going to be used and what that implies for your evaluation design.