A recently-released review that seeks to determine the reliability and usefulness of Los Angeles’ new value-added teacher evaluation system, has found significant limitations in how the results obtained using the system, can be used. The review, authored by Catherine Durso of the University of Denver, titled “An Analysis of the Use and Validity of Test-Based Teacher Evaluations Reported by the Los Angeles Times: 2011,” said the value-added model used by the LA Times to gets its results produced so-called probability bands where teacher results could have landed, instead of a single number assigned by the Times. Durso’s work was produced in partnership with the National Education Policy Center and funded by the Great Lakes Center for Education Research and Practice.

The single-scale the newspaper used to rate the effectiveness of teachers drew the greatest concern, “particularly the effectiveness classification categories published in the Teacher Ratings that the lay reader will be drawn to, and these are unreliable,” Durso wrote in her review.

She found that the potential for misuse of effective classification categories could prove to be harmful.

Still, the model used by the paper this year was an improvement over the VAM used to produce its rankings in 2010. More than half of the teachers rated in 201 had less than three-year’s worth of historical teaching data, and more than a quarter had been teaching for the first time. The larger amount of historical data available to input into the system, the more accurate is the rating it can produce.

Although, with enough data, the model provides a good estimate, it is singularly badly suited to provide a single number since it must account for a large amount of impression and inconsistency in testing results.

Even when multiple years of data are employed, the lack of predictive power limits the usefulness of the estimates to a parent considering whether an assigned class will improve a child’s test-based performance,” Durso stated.

The model and the five rating categories used by the paper both in 2010 and 2011 was developed independently by Dr. Richard Burton, a senior economist from the RAND Corporation. The Los Angeles Unified School District uses the name Average Growth Over Time to refer to this value-added model.

Although the LATimes releases the ratings together with a caveat that they are only estimated, there’s a risk of parents being misinformed and teachers’ careers being negatively impacted. This potential for misuse is why Durso felt that releasing the results might, on whole, be harmful rather than helpful to the public seeking insight into the quality of the district’s instructional staff.