The (Secret) Document That Drives Standardized Testing

by Richard P. Phelps In the excellent film The Insider, Jeffrey Wigand, an ex-tobacco company executive, faces a dilemma. In return for a severance package and the health insurance coverage it provides his family, he signs confidentiality agreements promising not to reveal the company’s research effort to boost the addictive power of cigarettes. When it [...]

by Richard P. Phelps

In the excellent film The Insider, Jeffrey Wigand, an ex-tobacco company executive, faces a dilemma. In return for a severance package and the health insurance coverage it provides his family, he signs confidentiality agreements promising not to reveal the company’s research effort to boost the addictive power of cigarettes. When it appears that he is preparing to speak to journalists anyway, tobacco company-contracted PR hacks assassinate Wigand’s character in the national media, and local thugs threaten his family’s safety. In the end, Wigand strikes the match that blows up tobacco industry deceit on CBS’s Sixty Minutes televised investigative news program.

I was reminded of Wigand’s story recently when a testing industry executive warned me not to reveal the specifics of a secret document currently being written—a document that, in my judgment, will effectively embed the findings of fraudulent, biased research in educational testing into US law. Among the several nasty effects should be an enormous waste of taxpayer dollars on millions of new and worse-than-worthless “audit tests”. The number of tests administered to our elementary-secondary students could double in some areas, but the quality of the results available from all tests will deteriorate.

Though this document will profoundly affect all Americans, whether directly involved in education or not, you cannot see it before it is published in its final, legal form, as a fait accompli in early 2013. I and perhaps a few hundred other testing aficionados read an early draft in 2011 but, legally, we cannot show it to you. We all signed confidentiality agreements.

Education insiders are currently writing in secret what is arguably the single most influential document in US education and psychology. Last updated in 1999, the Standards for Educational and Psychological Testing is being revised and, if on schedule, will be presented in its completed form to the public in early 2013. (The testing Standards should not be confused with more common, and far more public, content standards, a.k.a. curriculum).

You are forgiven if you did not know. In the spring of 2011, at a meeting of testing and measurement professionals, a couple of hours were set aside for all the “public comment” that would ever be allowed on drafts of the revision—well over 300 pages of mostly technical jargon. But even that peephole of transparency was advertised only within the incestuous world of testing experts, a.k.a. psychometricians.

I read most of those draft pages and even testified for the maximum allotment of six minutes. But, it hardly matters; in my experience, feedback is routinely ignored in the insular world of educational testing policy. The character and content of the new Standards were determined years ago when the draft writers were selected, and nothing else matters.

The Standards is important because practicing psychometricians adhere to its prescriptions piously. Just as a carpenter or plumber justifies a work expense by citing the housing code, those developing or administering tests justify their actions by citing the Standards. Even more consequentially, the Standards carry the force of law. In most of the world, societies have incorporated “high-stakes” tests into their education programs as a matter of course, as ordinary and expected as streetlights or city parks. In this country, by contrast, where anyone can sue anyone else anytime over anything, tests with consequences tend to attract lawsuits.

American judges may be smart people, but they know little of statistics generally, much less item response theory, differential item functioning, test-retest reliability, or any of the hundreds of other technical topics peculiar to psychometrics. In the most celebrated test-related court cases of the past few decades, judges have consulted the Standards, assuming it to be an objective or, at least consensually accepted, code of professional conduct, like the local housing code.

By 2013, the Standards will be neither objective nor consensual.

Nominally, three professional associations write the Standards—ergo the drafting group is called the “Joint Committee”—the American Psychological Association (APA), the American Educational Research Association (AERA), and the National Council on Measurement in Education (NCME).  The subjective bias starts there in Standards governance. Despite the fact that psychologists invented standardized tests and have conducted most of the relevant research over the past century, education groups outnumber the single psychology group two to one, and so can manipulate Standards content to their liking. If education research has been corrupted by its vested interests, as many are convinced, one should expect the Standards to be corrupted to serve education’s vested interests as well.

Sadly for our country, the most expert and trustworthy professional groups are not involved in any aspect of the Standards update. Whereas most APA members are now practitioners—counselors, therapists, and administrators—most psychology researchers have joined the Association for Psychological Science (APS). Included among APS members are the testing and measurement world’s best and brightest. Most are professors working in university psychology departments, as independent in their research as education professors are not. Regardless, they have no seat at the drafting table.

I would argue that industrial-organizational (a.k.a., personnel) psychologists should have a direct role in drafting the Standards, too, given their expertise, the enormous quantity of relevant, quality research they have produced over six decades, and their objectivity. As it stands now, I/O psychologists are but one division—Division 14—within the psychologists’ APA and a single vote in Standards decisions. The APA has 54 divisions.

That “test users” such as governors, mayors, teachers, and the public likewise are unrepresented in the Standards drafting process is evident in the draft language. In a few of the draft chapters, test users are portrayed as ignorant, biased, venal, and untrustworthy. The insinuations are worst in draft chapter 13—the testing policy chapter—which is quite cavalier and insulting.

Ironically, the same industry insider who warned me against revealing the contents of the revised Standards draft has himself publicly asserted its colossal social and legal impact on US society. Yet, he defends both the secrecy and insularity of the current drafting process. Power to draft the Standards as they see fit has been divvied up among the chosen few on the “Joint Committee”, a baker’s dozen of education professors and industry insiders. This is an extraordinarily small number of people to essentially be writing our country’s testing law.

In the case of chapter 13, at most “2-3 persons” craft our country’s testing policy. Not only is this a tiny group in number, these particular persons represent a biased and extreme point of view. Read the revised Standards, though, and theirs is the only point of view you will be allowed to know. As they have for a few decades now, these folks arrogantly declare a cornucopia of contrary opinion and evidence nonexistent.

Some relevant links:  

Richard P. Phelps is the author of Standardized Testing Primer (2007) and other books about testing and the founder of the Nonpartisan Education Review. He lives in Asheville, North Carolina.

Career Index

Plan your career as an educator using our free online datacase of useful information.

View All

On Twitter

Troubled student loan provider Sallie Mae re-brands with Navient offshoot #education #highered #edchat

1 day ago

What are the real HS graduation rates in Texas? @TXbiz would like to know #edchat #education #edreform

1 day ago

At many schools and colleges, disabled students lack the #edtech needed to excel #edreform #education

1 day ago

On Facebook

Recent News