A guide to evidence based policymaking

Picture this: it’s 1932, the height of the Great Depression in the United States. The unemployment rate is over 20% and growing. American citizens are looking to the government for answers. Franklin Delano Roosevelt is running for the US presidency against incumbent President Herbert Hoover. 

On the campaign trail, FDR remarks:

“The country needs, and, unless I mistake its temper, the country demands bold persistent experimentation. It is common sense to take a method and try it: If it fails, admit it frankly and try another. But above all, try something.”

While many would have expected candidates to campaign on very specific policies, FDR decided instead to campaign on an alternative approach – experimentation and learning. His message was: we don’t know what works, so let’s try things. 

We can figure out what works and what doesn’t along the way and adjust our policies accordingly. We can measure both successes and failure, and this gathered evidence can and should inform policy design. 

In your role as a policymaker, you can implement and pilot policies in ways that allow for learning and adjust policy accordingly. Join us as we look at the intricacies of evidence based policymaking.

Evidence based policymaking refers to the method of policy development that consults facts and credible, relevant evidence to make decisions, over political opinion or theory. For example, a politician taking this approach may use scientific evidence to aid the development of a new healthcare policy, instead of ideological beliefs. 

Ultimately, the premise of evidence based policy and practice is that policy decisions should be unbiased and driven by community or country needs, rather than a desire to win the backing of voters. 

Policymakers following this method might use existing evidence to support policy development or conduct new research to bolster their arguments. 

Below we explore examples of how experimentation and evaluation is carried out in the education sector in countries across the globe.

The past two decades have seen massive improvements in school enrolment around the world. This is partly due to the United Nations Millennium Development Goals (MDGs), which mobilised action in the period 2000-2015 before being superseded by the Sustainable Development Goals (SDGs) in 2015. One of the MDGs was to achieve universal primary education for all girls and boys. Some of the achievements under this goal include: 

• an eight percentage-points increase in the primary school net enrolment between the years 2000 to 2015; 
• nearly 50% decrease in the number of out-of-school children, globally, from 100 million to around 57 million in the same period; 
• remarkable improvement in primary education in Sub-Saharan Africa since the establishment of the MDGs. 

But as critics have pointed out since, the MDGs targeted increasing enrolment only, not quality. Hence these global improvements in school enrolment have not been matched by a similar progress in learning. 

For example, in India, where school enrolment is high, only about 44% of Grade 8 students can do a division equation. Similarly, only about 75% of Grade 8 students can read a Level-2 book. Level 2 is defined as ‘reading with help’ where engaging stories, longer sentences and language play are used for developing readers who still need some help. This has led to calls for improving learning in schools. 

We see a similar trend in other countries around the world, but few have picked up the mantle of experimentation like Kenya.

In an attempt to improve learning in schools in western Kenya, policymakers in the early 2000s collaborated with researchers to implement policies that they expected to work, and, when a solution failed, they tried another. 

Solution 1: Flipcharts 

Hypothesis: Poor education may be due, in part, to the lack of proper school materials. 

Intervention: Provide schools with flip charts covering mathematics, geography, health and two charts for science (agriculture and general science), as well as a teacher’s guide for science. They could be used in more than one classroom each day. 

Findings: Despite a large sample size and two years of follow-up data, the estimated impact of flipcharts on student test scores was found to be close to zero.  

Solution 2: Textbooks 

Hypothesis: Again, poor education may be due, in part, to the lack of proper school materials. This time: textbooks. 

Intervention: Provide schools with additional official government textbooks (English, maths and science). Sharing textbooks is common in Kenya, and two or three students typically share a workspace. Hence, a 0.6 textbook per pupil ratio was provided in English and science, and a 0.5 ratio was provided in maths, giving nearly all students shared access to a textbook. 

Findings: No evidence that textbook provision increased average test scores, or that it reduced either grade repetition or dropout rates. 

Solution 3: Extra teachers 

Hypothesis: Poor education may be due, in part, to large class sizes. When primary education was made free, it caused an influx of new students with little or no preparation, which affected the overall quality and achievements of all students as teachers had to spend more time bringing everyone up to the same level. 

Intervention: The ETP programme provided funding to hire a local contract teacher to address classroom overcrowding. School committees were responsible for hiring the contract teachers and were free to replace or keep the original contract teacher based on performance. The contract teachers were paid approximately one-quarter of the salary of regular civil service teachers but had similar educational qualifications. 

Findings: Providing school committees with funds to hire an extra teacher on a short-term contract had a positive effect on learning. Test scores of students in ETP schools were on average higher than those of students in comparison schools.

The three case studies above are not so much about ways to improve education, but rather about a mindset to use evidence and evaluate policy to understand what works where. Politicians can’t predict the future and therefore don’t know for sure whether their policies will be successful or not; this is why evidence and evaluation should be crucial parts of the policymaking process. 

Roosevelt’s policy of experimentation helped the US recover from the worst depression in living memory. At the University of Oxford’s Blavatnik School of Government, we believe evidence is imperative to address the complex challenges facing policymakers in today’s world, which is why our online short course on Evidence in Public Policy equips policymakers with the knowledge and tools to implement this mindset of experimentation.

This article was originally published on Apolitical.