Last year, when the Organization for Economic Co-operation and Development (OECD) released the results of the 2009 edition of the Programme For International Student Assessment (PISA), Brazil had reasons to celebrate: it was one of the three countries that presented a consistent evolution on the international test. Between 2000 and 2009, Brazilian students improved 33 points on the exam.
But, at the same time, results gave policymakers a warning – we still occupied the 53th place in the ranking that lists the performance of 65 countries. What PISA results showed is not new: we're improving, but not as much as needed. Our challenges are not small: almost 46% of our children and teenagers live with less than US$ 170, we still have 14,1 million illiterates (9,7% of the total population) and 7 million elementary school students are not in the grade they should be (23,6% of Brazilian elementary school students).
Like American policymakers, Brazilians are searching plausible and efficient answers for one question: how can education be fixed? This series of reports aims to discuss one of the policies imported by Brazil from the U.S.: the accountability system. The policy was adopted in America in the 1980s, but it only arrived here at the beginning of the 2000s. The Brazilian scenario is, thus, incipient. But it already brings up the same kind of doubts Americans have with such policy: is accountability really effective to tackle the lack of quality in education?
It is important to make a first distinction: the accountability system that operates in Brazil does not, in any case, provide dismissal of teachers or closure of schools considered "ineffective". But, as it works in any country, Brazilian accountability policies depend on external evaluations to exist. The federal government created, in 2005, the Brazilian Test, a biannual test that evaluates all Brazilian middle school students (6th and 9th graders). Until 2007, the exam was used only for diagnosis purposes by schools – their results were not released to society. But in that year, the Ministry of Education settled the Index of Quality in Basic Education (Ideb), which is calculated by the product of Brazilian Test results with the index of approval of each school. In other words, each schools was given a grade from 0 to 10 and a goal to achieve – if school A had an Ideb of 3,3 in 2005, it should achieve 4,3 in 2007. With Ideb, the federal government induced a policy that only makes schools accountable for their results. The state of ParanÃ¡, in the South of Brazil, adopted this system in 2000. "School Bulletin" was the name of the program which released, only to schools, an index that combined tests results, socioeconomic context and the results of an internal evaluation done by principals. Although it was a low-stakes policy, it brought a consequence that is known in high-stakes programs: principals cheated on the internal evaluation because they were afraid to get fired.
But there are several states and municipalities that implemented an accountability system that relies on one of the most controversy policies in education: merit pay. Based on test results and absence index, teachers receive bonuses at the end of every school year. That is the case of Rio de Janeiro (state and municipality) and the states of GoiÃ¡s, Pernambuco and SÃ£o Paulo. In fact, the state of SÃ£o Paulo is re-shaping its policy because, since it was implemented, there were no real gains in tests results. In the last three years, only 5th graders have improved their scores – and the evolution is small. While results for Portuguese have not risen between 2009 and 2010, those students presented the following results for Math tests: 204,1 in 2009 and 204,6 in 2010.
On July, the municipality of SÃ£o Paulo inaugurated a new kind of accountability policy with merit pay basis with the creation of the Index of Quality in SÃ£o Paulo (Indique), a measure that which will take into account socioeconomic status of students, giving different weights to improvement in accordance with the reality of schools."We will not punish, but portray reality", said Francisco Soares, a professor at the Federal University of Minas Gerais (UFMG) and the index's creator.
Indique will be calculated with students' Portuguese and Math scores in the SÃ£o Paulo Test (Prova SÃ£o Paulo), that will be weighted by the level at which the school is and the improvement of students. The result will be multiplied by the "effort" showed by the school in order to improve. The factor "effort" takes into account the socioeconomic status of students, a value determined from a questionnaire answered by parents of students in the last three years. The higher the social level of school, the lower will the effort that their teachers will have to achieve a good performance and vice versa. Bonuses will be conceded by the achievement of goals related to Indique, but, according to the press office of the Secretary of Education of SÃ£o Paulo, these goals have not been settled yet.
The Academic Reaction
After a decade of the arrival of the accountability policy in Brazil, it's possible to say that the academic world in the country is still digesting its effects. There's no large-impact study that investigates how our educational system reacted to merit pay, for example. Researchers are divided. Last year, a group of educators that was present in the 33rd meeting of the National Association of Graduate Studies and Research in Education (Anped) created the "Movement Against Testing High Impact". Signed by 82 researchers, the movement aims, firstly, to alert parents, students, administrators and the government that "high-impact tests" are inaccurate. Besides that, they want to discuss the creation of a code of ethics for the development, implementation and use of data obtained by the evaluation process.
"It's an academic movement that will start a debate in a certain circuit", explains Ocimar Munhoz Alavarse, professor at the University of SÃ£o Paulo (USP) and one of the coordinators of the movement. Mr. Alavarse defends that the country faces a serious problem with accountability and evaluation: teachers and principals do not understand what tests scores mean. For example: if a student gets a 230 score in Brazil Test (Prova Brasil), what does it mean? What does he know? And how can the school improve its approach, in order for the student to learn? "There are two problems, actually: the misuse of assessments and the difficulty of intelligibility that teachers and principals face with tests results", he defends.
Marcio Costa, a professor at the Federal University of Rio de Janeiro (UFRJ), who is not a signatory of the Movement, is sympathetic to bonuses and accountability, but not for the country's present reality. "With salaries almost always unsatisfactory, bonus seems a patch badly made", he says. According to the Ministry of Education, the average monthly salary for teachers in Brazil is US$ 954,37. Mr. Costa advocates that, although tenure is a necessary safeguard against arbitrariness and persecution, it should be removed from education. "The end of tenure will give another meaning to bonuses. If good merit pay system is not followed by the ability to take effective measures against poor performance, something is wrong". He reminds that the academic world has much to contribute to the debate about accountability. "This discussion has to take into account our institutional culture. The initiatives are still too recent to have conclusive opinions", says Mr. Costa.
Fraud and Examples of Gaming the System
One of the most popular themes when it comes to the accountability policy are the cases of fraud on tests and the examples of teachers and principals who try to "game" the system. In Brazil, the latest case of fraud happened with the High School Examination (Enem), test taken by high school students to enter in federal universities. In 2010, the test was leaked to the press. In addition, Brazilian newspapers reported several cases of cheating. Another common problem is the lack of control over the students who actually participate in the Brazilian Test (Prova Brasil). As principals don't need to register absent kids, they tend to put only their best students to take the examination. In the absence of a measure that indicates the number of missing students, the result rises.
Recently, the municipality of Rio de Janeiro also faced accusations of "gaming" their accountability system. A report published in June by the magazine EducaÃ§Ã£o revealed that the formula used to calculate bonuses was not clear. Finally, there are those who say that accountability policies are causing a reduction in the school process, as teachers, concerned about the bonus they can receive, teach only what the test asks. According to Ocimar Munhoz Alavarse, professor at the University of SÃ£o Paulo (USP), frauds and "gaming" happen because policymakers tie many important decisions to tests, like bonuses. "Tests could not have the weight they do. The results are not supposed to be âthe' measure of quality, but one of various measures", he explains.
Beatriz Rey received a degree in journalism from Casper Libero College). She is the editor of EducaÃ§Ã£o , Brazil's education publication of record, and volunteers as a teacher of adult literacy. You can follow Beatriz on Twitter, connect with her on LinkedIn, and read more on her website.