Education Software is Booming — But is the Quality There?

A 2010 federal review of  Carnegie Learning’s flagship software, Cognitive Tutor, said the program had “no discernible effects” on the standardized test scores of high school students. However, Carnegie’s Web site makes no mention of that — which suggests that many companies ignore well-regarded independent studies that test their products’ effectiveness, write Trip Gabriel and Matt Richtel at the New York Times.

The review, carried out by the Education Department’s What Works Clearinghouse, found only four of the 24 studies analyzed of Cognitive Tutor’s effectiveness met high research standards.

So are some firms misrepresenting research by cherry-picking results that lack the scientific rigor required by authorities?

“The advertising from the companies is tremendous oversell compared to what they can actually demonstrate,” said Grover J. Whitehurst, a former director of the Institute of Education Sciences, the federal agency that includes What Works.

School officials, confronted with a morass of complicated and sometimes conflicting research, often buy products based on personal impressions, marketing hype or faith in technology for its own sake, writes Gabriel and Richtel.

Carnegie, one of the most respected of the educational software firms, is hardly alone in overpromising or misleading. Similarly, Pearson’s website cites several studies of its own to support its claim that Waterford Early Learning improves literacy, without acknowledging a 2009 study’s conclusion that it had little impact.

 “To compare public relations analysis to a carefully constructed research study is laughable,” said Alex Molnar, professor of education at the National Education Policy Center at the University of Colorado. “They are selling their wares.”

Carnegie officials say 600,000 students in 44 states use its products, many taking teacher-led classes three times a week with Carnegie-provided workbooks and spending the other two class periods in computer labs using Cognitive Tutor, writes Gabriel and Richtel.

“The full curriculum can cost nearly three times as much as a typical textbook over six years.”

Steve Ritter, a founder and the chief scientist of Carnegie Learning, said there were flaws in the What Works Clearinghouse evaluations of Cognitive Tutor.

“What you want to focus on is more of the why,” he said, “and less of a horse race to find out what works and doesn’t.”

Gregory W. Capelli, co-chief executive of the Apollo Group, which runs the 400,000-student University of Phoenix and bought Carnegie Learning this summer, said his company first ran its own pilot project with the software and also examined independent research, writes Gabriel and Richtel.

 “My daughter, who’s in eighth grade, used this product,” Capelli said. “She would do very well” in some lessons “and not in others… What I liked about it is that once she got it, it would allow her to go on to the next part of the tree.”

A Carnegie spokeswoman, Mary Murrin, said in a statement that the company used “the data from all studies with varying outcomes to continuously improve our programs.”