Gallup, 21st Century Skills and the Education Technology Cargo Cult

by Matthew K. Tabor

When you ask education professionals what “21st Century Skills” are — such as attendees at last week’s International Society for Technology in Education (ISTE) 2013 convention in San Antonio, or the National Alliance for Public Charter Schools conference currently underway in Washington — you get a hundred different answers. How we use technology in schools varies, and ideas about how education can benefit from technology vary even more. We’re experimenting, analyzing, adjusting and making predictions about how it will change, and trying to do it all on a sensible budget.

Matthew K. Tabor

But when you ask education technology providers about 21st Century Skills, the answers are clear: We know what they are, we know why they’re useful, and, by the way, we just happen to know where you can buy everything you need to develop them!

A survey by Gallup, Microsoft Partners in Learning and the Pearson Foundation called “21st Century Skills and the Workplace” drew a bold conclusion: “What Works in Schools is Real Work.” In an article on Education Week’s Global Learning blog, Executive Director of Gallup Education Brandon Busteed complemented his presentation to the Asia Society’s Partnership for Global Learning Conference with an explanation of the survey’s findings:

The best type of curriculum for preparing students for the workforce is one that focuses on real-world problem-solving. It sounds simple, but for the first time, we have clearly established a link between students learning 21st century skills and future work success.

It does sound simple, but that’s not the important part. The problem is that Gallup doesn’t have the data to back it up.

Telephone interviews with 1,014 students and workers aged 18-35 posed a series of questions about how they related workplace performance, happiness, and preparedness with their 21st century skills experienced in school. Busteed claims a link between the development of 21st century skills in school and “higher work quality”:

The results of a Gallup/Microsoft Partners in Learning/Pearson Foundation study show that young workers in the U.S. who reported learning 21st century skills in their last year of school are more likely to say they have higher work quality. In fact, those reporting high levels of 21st century skill development in school are twice as likely to have higher work quality compared with their peers who had low 21st century skill development.

Mr. Busteed, we need to have a conversation about causation vs. correlation before school districts throw millions more dollars at the companies who funded your survey.

It shouldn’t be a surprise that students who encounter technology, communication strategies and their required skills more often in school go on to better careers. Technology comes from funding; more funding means wealthier school districts and higher-quality schools; wealthier school districts are populated by family environments that foster better educational achievement.

There is also no distinction made by Gallup between technology purchased and used in the home for academic purposes and technology exclusively delivered by schools, nor between 21st century skills provided uniquely by schools and the acquisition and application of relevant skills from the outside.

Technology integration and use correlates with an environment that produces achievement, but Gallup has failed to isolate technology as the variable that drives that achievement.

Busteed continues:

Positive responses to the following two items have the strongest link to work quality:

“Worked on a long-term project that took several classes to complete”
“Used what you were learning about to develop solutions to real-world problems in your community or in the world”

Long-term projects that require several classes to complete take collaboration and communication school-wide — which is commentary on that school’s quality and culture more than that they derived specific benefits from working on A Really Big Project. Again, Gallup’s data hasn’t isolated technology use or 21st century skills uniquely cultivated in schools as factors that result in higher achievement.

The survey, which you can read in full [PDF], also suffers from several methodological problems:

  • Questions were posed to respondents in terms of “21st Century Skills,” defined by Gallup as “collaboration, knowledge construction, skilled communication, global awareness, self-regulation, real world problem solving, and use of technology for learning.” These terms are highly subjective, and asking a respondent to consider and analyze them on a moment’s notice in relation to their academic and professional careers yields a poor result.
  • Retrospective self-reporting, as a cognitive psychologist critical of the survey pointed out, carries little weight — and especially when the considerations are so subjective.
  • Respondents’ work experiences vary in both scope and perspective over the 18-35 demographic.

Gallup’s conclusions about the statistical correlations between 21st Century skills and workplace success are also weak, as the tables on pages 18 and 19 of the PDF show. Their argument simply is not as strong as Busteed would have you believe.

But the real harm in Gallup’s report is in the suggestion that the presence of technology and nebulous 21st century skills in schools will lead to higher academic achievement, greater happiness and a stronger workforce.

We’ve seen this sort of thing before.

The Melanesians, who populate islands from New Guinea to Fiji, were stunned during World War II by supplies coming in from Japanese airdrops on their islands — food, clothing and tools far more advanced than the Melanesians had ever seen. When the Americans occupied the territory, even more supplies came from the air.

It seemed simple: if the Melanesians could replicate the airstrips, structures, technologies and behaviors of the Japanese and American militaries, supplies would surely follow from the sky. They made radios out of coconuts, cleared out crude airstrips, and marched like soldiers with sticks in place of guns because they thought the rituals would attract goods. Supplies didn’t come, and the Melanesians realized what Gallup hasn’t: that causation and correlation are two different things.

Cargo cults, first documented in the late 19th century, continue exist in many sectors including education and its ed tech subset. But instead of coconut radios stretches of island swept free of rocks, the Gallup, Microsoft and Pearson cargo cult encourages schools to erect SMARTboards, wave around tablet computers, and meddle in software as they promise that high academic achievement and successful careers will follow — and they’re quite happy to connect you with a representative who can speak to you about solutions to your district’s individual 21st century needs.

The 2013 “21st Century Skills and the Workplace” survey delivers wonderful news for technology vendors. They get public dollars year after year to play out education technology cargo cult rituals that entail peddling a mix of technology solutions and a sense of community humbly pitched by Busteed as, “a silver bullet for solving many of the world’s problems.”

Unfortunately, that means teachers, taxpayers and especially students continue to wait in the village for supplies they’ll never receive.

Matthew K. Tabor is the editor of EducationNews.org. He can be reached at editor@educationnews.org.

Matthew Tabor

Matthew Tabor

Matthew is a prolific, independent voice in the national education debate. He is a tireless advocate for high academic standards from pre-K through graduate school, fiscal sense and personal responsibility. He values parents’ and families’ rights and believes in accountability for teachers, administrators, politicians and all taxpayer-funded education entities. With a unique background that includes work in higher education, executive recruiting, professional sport and government, Matthew has consulted on new media and communication strategies for a broad range of clients. He writes the blog “Education for the Aughts” at www.matthewktabor.com , has contributed to National Journal’s ‘Expert’ blog for Education , and interacts with the education community on Twitter and Google+.