(Mis)Understanding the NAEP Results

3.27.10 – by Chad Aldeman – Stories from the NY Times, Mother Jones, and the Washington Post bemoaned the flat National Assessment of Education Progress (NAEP) reading scores released Wednesday. Jay Matthews called it the epitaph of the No Child Left Behind era. The results aren’t quite so simple.

(Mis)Understanding the NAEP Results

Stories from the NY Times, Mother Jones, and the Washington Post bemoaned the flat National Assessment of Education Progress (NAEP) reading scores released Wednesday. Jay Matthews called it the epitaph of the No Child Left Behind era. The results aren’t quite so simple.

See, NAEP is different than most standardized tests. It takes a sample of the current population in every state, so this year’s population of kids is compared to the last time the test was administered. There’s an automatic correction for changing demographics, so as America has gotten less white, so has NAEP. In statistical terms this creates something called Simpson’s Paradox, which makes trend lines seem worse than they really are because of a hidden variable, in this case, race (Matthew Yglesias touched on this point yesterday).

To show how this impacts NAEP scores, here are the results of the long-term trend NAEP results for fourth-grade reading from 1975 to 2008 (I’m using the long-term trend version of NAEP, because it’s been largely unchanged since its first administrations in the 1970s. Its had one significant format change, in 2004, when NAEP administered both the new and old formats. Hence the dotted and solid lines in all of the following graphs). As the chart below shows, average fourth-grade reading scores have risen only modestly, from 210 in 1975 to 220 in 2008.

NAEP01

This is basically the same thing that showed up in yesterday’s results. There’s been some small gains over time, but year-to-year progress has been small.

But that’s not the whole story. See, these overall trend lines are a sampling of America. As we’ve become more diverse, NAEP has changed its sampling ratios to reflect our changing society. This chart shows the percentage of students drawn from racial/ ethnic categories over time. In 1975, NAEP test-takers were 80 percent white. By 2008, only 56 percent were.  There were three percent more blacks in 2008 than in 1975, and Hispanics had quadrupled from five to 20 percent.

NAEP02

So, because NAEP has gradually included more black and Hispanic students, and black and Hispanic students score lower, on average, than white students, the total score doesn’t reflect the true gains made by each group. The chart below shows scores taken from the same testing years, this time disaggregated by race.

Each group has actually made greater gains over time than the overall total. White students increase 11 points, one more than the national average. Black students scored 23 points higher, and Hispanic students were scoring 24 points higher in 2008 than they were in 1975 despite quadrupling in size. In other words, the white-black and white-Hispanic gaps are closing and every group is scoring higher, but the national score is showing more modest improvements because of demographic changes.

NAEP03

This is an important distinction to make, because it means the test score results are not just a matter of classroom teaching and learning (to be completely clear, I don’t think NAEP results can be easily attributed to national education policies like NCLB either). The students themselves have changed in important ways, and to break even or to make small achievement gains as society becomes more diverse is an accomplishment worth celebrating. At the very least it’s worth understanding.

For more background reading on NAEP, try Education Sector’s NAEP Explainer.

Comments


  1. VanceK

    Thank you, Chad. Isn't it interesting how people (even Jay Matthews) are so quick to use a surface interpretation of the data to support their agendas. I'm still wondering how great the uproar would have been if the Bush administration had signaled their intent to implement even half of what President Obama and Secretary Duncan have put into place.

    Change is hard…but necessary, and NCLB served as the lever to overcome a huge amount of intertia and get the ball rolling. Now it is up to use to maintain and grow that momentum. Not everyone will be happy.


  2. Robert

    Maybe true. But, the sad thing is when we compare our children's scores to that of other nations. That is where we fail. We also need to convince more students to study science, math, and engineering degrees in college. We have a lot of work to do. We need more people in the trenches working with these kids. Even if we made huge gains, we could still make more.

Leave a comment

Saturday

March 27th, 2010

Jimmy Kilpatrick

On Twitter

Camden's high #education spending and low graduation rates raise eyebrows http://t.co/U9YtvhVGyV #education #edchat #edreform

24 hours ago

Data shows UK students may be repaying loans well into their 50s http://t.co/7wnUt4oDNU #edchat #highered #education

1 day ago

Survey shows link between strong schools, teachers undeniable http://t.co/cokkcLQPpl #edchat #education

1 day ago

On Facebook

Subscribe

Enter your email to subscribe to daily Education News!

Hot Topics

Career Index

Plan your career as an educator using our free online datacase of useful information.

View All