Does pay-for-performance in education work at scale? Evidence from a nationwide program in Peruvian secondary schools: Guest post by María Lombardi

The World Bank
Development Impact Guest Blogger



Teacher quality is key for student achievement (Hanushek and Rivkin, 2010; Rockoff 2004). Individuals exposed to better teachers perform better in school, and are more likely to attend college and earn higher salaries (Chetty et al., 2014). However, the payment schemes in most educational systems are based on lifetime job tenure and flat salaries that are usually linked to teachers’ seniority, providing few incentives for excellence in teaching. Academics and policymakers have proposed tying teachers' pay to their students' performance in an attempt to increase teacher motivation, accountability, effort, and ultimately student learning.

While pay-for-performance programs in education have been implemented in several developing countries such as India, Kenya, China, Chile, Brazil, Mexico, and more recently in Peru, evidence on their effectiveness is scant and inconclusive. My job market paper, coauthored with Cristina Bellés-Obrero, addresses this question in the context of Bono Escuela (BE), a nationwide teacher pay-for-performance program implemented in 2015 in all public secondary schools in Peru.

BE is a collective teacher incentive in which all public secondary schools compete within a group of comparable schools on the basis of their annual performance. There are 395 groups in total, and the average BE group has 35 schools. Every teacher and the principal in schools ranked in the top 20% within their group obtain a fixed payment amounting to over a month's salary. The incentives provided by the BE are collective (at the school- level), as all teachers are rewarded if their school wins, although the main performance measure used to rank schools is their average score in a 2015 nationwide math and language standardized test, taken only by 8th graders. This feature of the program, which we exploit in our identification, implies that a school's probability of obtaining the bonus hinges on the achievement of 8th grade students in 2015.

Identifying the causal effect of BE on student achievement
We rely on a novel administrative database provided to us by the Peruvian Ministry of Education, which covers the universe of students in 2013-2015. There were 2.4 million secondary school students in Peru in 2015, 1.7 million of which attended a public institution. This database contains annual information on the grades awarded to students by their teachers in every subject (their “internal grades"), which we use as our outcome variable. Internal grades have the advantage of capturing the skills of students which are targeted by the program (i.e., their basic competencies in math and language) without directly influencing teachers’ likelihood of obtaining the bonus, thus making them less prone to manipulation. Importantly, teachers' grading tactics should not be influenced by the incentive, since internal grades have no direct impact on a school's BE score. The availability of achievement measures for students in all grades allows us to compare the changes in internal grades of 8th graders to those of 9th grade students attending the same school, before and after the incentive was introduced, providing difference-in-differences estimates of the effect of BE on student achievement. Using data on the overlap of teachers in 8th and 9th grade, we assess whether the BE had an impact on student achievement in our comparison group, and discard the existence of any spillovers which could bias our estimates.

BE had no impact on student achievement
Despite the fact that the incentive was large, BE had no impact on students' math and language internal grades. Our coefficients are precisely estimated, allowing us to reject effects larger than 0.008 standard deviations (SD) in math, and 0.017 SD in language, well below the treatment effects found in studies such as Muralidharan and Sundararaman (2011), and Contreras and Rau (2012). When separately examining the impact of the program in each of the 395 BE tournaments, we find little variance around the zero average effects (Figure below), providing additional evidence of the null effect of BE on student achievement.



Due to the tournament nature of the BE, the incentive could be impacting schools closer to the margin, and leaving sure-winners and sure-losers unaffected. We explore whether there are differential effects across these and other dimensions of the incentive, and do not find evidence of heterogeneous effects.

Why didn’t student learning increase?
In order to answer this question, we carried out an online survey of a sample of public secondary school teachers. We provide suggestive evidence that the null effect is not a result of the size or collective nature of the incentive, or driven by teachers being uninformed about the BE, or only focusing on increasing standardized test scores – the incentivized outcome – without influencing their students' learning in a meaningful way. We argue that certain features of the standardized test linked to the bonus might have hampered teachers' ability to boost student performance in terms of this measure, potentially discouraging them from exerting higher effort:

1. Since the standardized test tied to the bonus was implemented for the first time in secondary schools in 2015, teachers might not have known what pedagogical practices result in higher test scores. The fact that students had no stakes in these evaluations might also have played a role in weakening the mapping between teachers' effort and their chances of winning the bonus, as suggested by the findings of a teacher performance pay program in Mexican high schools (Behrman et al, 2015).
2. The incentive might also have been diluted if schools were uncertain about their potential ranking within the group of schools they were competing against. Given that they had no prior experience with the standardized test tied to BE, this is not unlikely.
3. Teachers might not have had enough time to react to the incentive. The program was announced 4 months before the test, and while this is a considerable amount of time, schools had 7-8 months to react in other studies finding positive impacts. We thus cannot discard that this one of the reasons why BE had no effect.

In ongoing work, we examine the medium-run impacts of BE by looking at students’ performance in the second year of the program, where teachers will have acquired more experience with the standardized test and the BE program, and thus some of the potential issues that could be diluting the effect of the incentive program may disappear. Extending the program could also result in schools devoting higher effort to improving the learning of 7th grade students, in anticipation of their participation in the following year’s standardized test. However, the program could have undesirable long-run impacts if teachers become more acquainted with how to teach-to-the-test, or if schools divert resources away from students not reached by the standardized test tied to the bonus.

Policy implications

Most of the existing studies on teacher pay-for-performance tackle this topic using a randomized control trial. While these experimental studies make important contributions towards understanding whether performance pay is an adequate tool for increasing student achievement, their findings are not necessarily generalizable to a scaled-up program. This notion is put forward by Banerjee et al, 2016, who show that a successful educational intervention led by a NGO did not yield the same initial impact when it was scaled up and implemented within the existing educational system. Budgetary constraints (Kerwin and Thorton, 2015) and opposition from teacher unions (Mizala and Schneider, 2014) make several aspects of these types of interventions unfeasible in a nationwide program. It is thus crucial to better understand the role played by the features of teacher pay-for-performance programs in their success.

María Lombardi is a Ph.D. candidate in Economics at Universitat Pompeu Fabra. More details about her research can be found on her webpage.