Daniel Koretz is the author of the recent book "Measuring Up: What Educational Testing Really Tells Us", published by Harvard University Press. The book examines the issue of testing, annual yearly progress (or lack thereof) and the entire issue of testing, and what information we can really glean from standardized testing, formal testing, informal testing and observational testing. These issues are so complex and intricate that every few weeks in the near future, Professor Koretz will be responding to some profound questions dealing with the field of psychometrics and what all this testing really tells us, and what it FAILS to tell us and how the testing process is being foiled by some teachers. Today's column will focus on what Professor Koretz labels "the myth of the vanishing variance".

1) Professor Koretz, let's talk about this concept of the "myth of the vanishing variance". First of all,what do you mean by the " myth of the vanishing variance".

"Variance" is just one statistical term for how variable or different students are.The "myth of the vanishing variance" is the mistaken notion, common in policy circles, that we can greatly reduce the variability of student performance.We certainly can reduce it, but much less than some people expect.

2) Certainly, we would like to see the left hand side of the bell shaped curve diminished or eliminated, but does that data really support this- is "mental retardation" really being eliminated?

There is no doubt that some students at the lower end of the distribution shouldn't be there, that they are not manifesting their potential.Some are ill, others are abused, some are taught poorly, others are discouraged by peers or even their parents from putting effort into school, and so on.Certainly, one of our main goals should be to try to reduce this and to give all students a better chance to manifest their potential. However, even if we were to do better at this, it would not make the overall variation in performance decrease greatly. I'll mention three reasons.One reason is that schools remain only one influence on student achievement.

A second reason is that there are also students higher up on the distribution who are not manifesting their potential, often for the same reasons, and they too will move up if given more support.That would maintain some of the differences among these students.

Finally, although it is not often discussed, people simply differ in their capabilities. I'm a klutz and a slow runner, while my daughter is a marathoner.Much of that difference would remain even if we trained the same.Similarly, some students will learn academic subjects more rapidly than others.You mentioned mental retardation, but variations in students' capabilities are important all along the distribution, not just for students with cognitive disabilities.

3) For our readers, how would you define homogeneity and heterogeneity, and how does thisrelate to the " myth of the vanishing variance"?

Homogeneity and heterogeneity simply refer to the similarity or variability of people, respectively.For example, if you look at the height of elementary school students, a group of first graders will be more homogeneous (less heterogeneous) in height than the entire school body.When people say that a school is heterogeneous in terms of student background, they are only saying that the school contains students with highly varied backgrounds. Heterogeneity of student performance on tests appears to be pretty much a universal—it appears, for the most part with relatively minor variations, in every context where we have looked.

4) Now, you and I both know, that no matter how much money the government throws at schools that we are still going to have approximately 68 percent of the population in the " average " range or within one standard deviation on both sides of the mean.  Is it unrealistic to think that we can encourage, push, cajole, and assist all of that group of one standard deviation below the mean—can we really push them all above average?

We can never push all students above the average—the average will just move up with them.We could, with enough political will and resources, move a great many students above our current average, but it would take a lot of work and a good bit of time.

5) Vygotsky and a few others have indicated that with help and assistance and mentoring and tutoring that we can assist people to move up in their " zone of proximal development". How much can one student in that average range –in that middle 68 percent of the bell shaped curve- how much can they be moved to the right?

As you learn more, your zone of proximal development will move with you.Many people can perform Brahms symphonies well, but in almost all cases, they had to study for years before their zones of proximal development encompassed Brahms.

A different way to pose this question is this: just how fast can we expect the curve to move to the right?We have a good bit of historical data about this.On that basis, I would say that if you are looking at a very large group—say, a whole state or the nation—an increase of about .04 or .05 standard deviation a year is pretty darn good.Anything much faster than that makes me worry about possible score inflation.

6) In terms of your concept of the "myth of the vanishing variance", we have been under No Child Left Behind for several years. Is there any proof that the variance is indeed vanishing, or has it stayed the same, or are we seeing more and more variance and heterogeneity?

The National Assessment of Educational Progress results do not suggest much consistent change in the spread of student performance.In mathematics, for example, the standard deviation of scores (just another measure of variability, technically the square root of the variance) has shrunk a very small amount in the fourth grade but has fluctuated inconsistently in the 8^th and 12^th.This means that there have not been consistently and substantially different trends in performance for kids in the top and bottom of the distribution.

7) There is some overlap between political decisions and what happens in the average classroom. If we allow larger and larger numbers of immigrants and illegal aliens into the U.S. what will happen to the variance – will the heterogeneity increase? Stay the same?

Immigration can increase the overall heterogeneity of our student population, but less than you might think.One reason is immigrants are themselves highly heterogeneous in achievement.Some are low achievers, but some are high achievers.Immigration is more of an issue for individual schools or districts.Any rapid influx of new students—whether immigrants or just students from elsewhere in the U.S—can make the population of a school or district much more heterogeneous.

8) Is there really any way to diminish the variance on the lower left end of the bell shaped curve? (Surely we do not want to diminish the variance on the upper right)

When I wrote about the myth of the vanishing variance, I was trying to help people to set realistic goals. We can't eliminate the great variability in performance.However, we can—and certainly should—do a great deal to help low achieving students do better. What we need to do is to set reasonable targets, put the needed resources in place, and gradually ratchet up our expectations.

Published October 9, 2008