The National Reading Panel: A Comprehensive Guide to Commonly Cited Criticisms

The National Reading Panel: A Comprehensive Guide to Commonly Cited Criticisms
EducationNews.org
by Chandra Keller-Allen

Table of Contents

Section

Page

The Basic Premise

1

  The State of Reading Achievement in our Nation

1

  Three Basic Premises

3

Panel Composition

4

  Panel Member Selection

4

  Panel Member Qualifications

5

  Panel Member Knowledge with Respect to Meta-Analysis

8

  Conflicts of Interest

9

  Panel Member Roles

10

Methodology

12

  Study Selection

12

  Qualitative Research

12

  Consideration of the Inclusion of Qualitative Research

13

  Selection Criteria: Peer-Reviewed Journals

14

  Data Analysis

16

Topics Covered by the NRP

21

  Topic Selection Process

21

  Importance of Topics Not Included in the NRP Report

23

Phonics Instruction

26

  The Issue with Phonics Instruction

26

  The Basis of the NRP Report Phonics Findings

29

  NRP Findings on the Effectiveness of Systematic Phonics Instruction

30

  Findings in Terms of Effect Sizes

31

  NRP Findings for Particular Subgroups

33

  Inclusion of English Language Learners in the NRP Analysis

35


Section

Page

  NRP Findings on Systematic Phonics Instruction and Reading

  Comprehension

35

  NRP Findings on Systematic Phonics Instruction and Spelling

40

  NRP Findings on the Role of Teachers in Systematic Phonics

  Instruction

40

  Systematic and Explicit Phonics Instruction Does Not Equal "Skill and

  Drill"

41

References

43

Tables and Charts

Page

Table 1 2003 Grade 4 NAEP Reading Achievement Levels, in Percents

2

Table 2 2003 Grade 8 NAEP Reading Achievement Levels, in Percents

2

Figure 1 Grade 4 NAEP Reading: Percent of Students Scoring Below Basic

2

Table 3 Comparisons by Reading Ability

34


The National Reading Panel: A Comprehensive Guide to Commonly Cited Criticisms

The Basic Premise

The State of Reading Achievement in our Nation

The results of the 2003 National Assessment of Educational Progress [NAEP] starkly remind us of the state of reading achievement in our country.  These data are not an anomaly but rather follow a trend line of several years (see Chart 1 below).  NAEP results are reported, as authorized by law, by student achievement levels.  The levels and their definitions are as follows: [1]

1. Basic - This level denotes partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade.

2. Proficient - This level represents solid academic performance for each grade assessed.  Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real world situations, and analytical skills appropriate to the subject matter.

3. Advanced - This level signifies superior performance.

Table 1 and Table 2 show the 2003 NAEP reading achievement results for 4 th and 8 th graders as percentage of students scoring at each achievement level, including those scoring below the basic level.  Almost 40 percent of all fourth graders and almost 30 percent of all eighth graders are scoring below the basic level on the NAEP assessment, indicating that a large group of children are not displaying even partial mastery of grade level knowledge and skills in reading.  The rates of children scoring below basic are considerably higher for African American, Latino, and poor students, demonstrating a persisting achievement gap.


Table 1

2003 Grade 4 NAEP Reading Achievement Levels, in Percents

Group

Advanced

Proficient

Basic

Below Basic

All

7

23

32

38

African American

2

10

27

61

Asian

11

26

32

31

Latino

2

12

29

57

Native American

2

14

31

53

White

10

29

35

26

Poor

2

13

29

56

Non-poor

11

30

34

25

Source : The Education Trust http://web.archive.org/web/20061003224949/http://66.43.154.40:8001/projects/edtrust/index.html

 

Table 2

2003 Grade 8 NAEP Reading Achievement Levels, in Percents

Group

Advanced

Proficient

Basic

Below Basic

All

3

27

42

28

African American

0

12

41

47

Asian

5

33

40

22

Latino

1

13

40

46

Native American

1

17

41

41

White

4

35

43

18

Poor

1

14

41

44

Non-poor

4

35

43

18

Source : The Education Trust http://web.archive.org/web/20061003224949/http://66.43.154.40:8001/projects/edtrust/index.html

     Figure 1 - Trends remain relatively stable over time

        Source : The Education Trust http://web.archive.org/web/20061003224949/http://66.43.154.40:8001/projects/edtrust/index.html

Despite these alarming statistics, some critics claim that our nation's current educational efforts with respect to reading are sufficient.  For example, as cited in Shanahan (2004, pg. 255) Allington (2002, pg. 16) states, "I'll grant American schools could be improved, and that we could improve children's reading proficiency, but it seems to me that it's almost time for a national celebration of what we have accomplished up to this point."  It is disconcerting that the 2003 NAEP scores in reading achievement, or the results from any of the four previous assessments, presented here could be cause for any celebration.

Three Basic Premises

In 1997, Congress requested the creation of a National Reading Panel [NRP] to assess the current state of reading research and bring this information to the attention of educators, policy makers, and parents.  Shanahan, one of fifteen original Panelists, elucidates three basic premises for the work of the National Reading Panel:

The NRP was fundamentally based upon the idea that schools should try to improve children's reading abilities and close the achievement gaps between rich and poor, black and white, and so forth.  Another basic premise of the NRP was that research could provide valuable direction toward improving achievement and that it was thus necessary to make an objective public determination of the scientific research evidence on reading.  A final premise of the NRP was that the research evidence had to be analyzed in a manner consistent with the highest standards for research synthesis to limit the influence of personal belief, self-interest, and other biasing factors (Shanahan, 2004, pg. 255).

The Panel conducted its work under the assumption of these premises.  The following sections provide the reader with considerable information and detail on areas of common critique with respect to the National Reading Panel and its report.  While not exhaustive, this document demonstrates that many of the criticisms are unfounded or based on misinformation.  Ideally, rather than spend precious time and energy debating these critiques, our nations educators, administrators, and policy makers should truly be about the business of improving reading instruction for the great numbers of students who are struggling, as evidenced by NAEP scores.  Perhaps if we address these criticisms comprehensively and in depth, we can then truly focus on this shared goal.

Panel Composition

Panel Member Selection

The original Congressional charge to constitute the panel required the Secretary of Education and the Director of NICHD to work together to select members of the Panel.  The Congressional charge states:

Noting the fact that the NICHD is already collaborating with the Department of Education, the Committee urges the Director of the NICHD in consultation with the Secretary of Education, to convene a national panel to assess the status of research-based knowledge, including the effectiveness of various approaches to teaching children to read. The Committee recommends that the panel be comprised of 15 individuals, who are not officers or employees of the Federal Government and include leading scientists in reading research, representatives of colleges of education, reading teachers, educational administrators, and parents (Senate Report 105-58, 1997).

The panel selection was completed in accordance with the Congressional charge.  Camilli, Vargas, and Yurecko (2003) raised the issue of the transparency of the panel member selection process.  The authors write,

The NICHD and Secretary of Education appear to have conducted a selection process consistent with the IOM [Institute of Medicine] guidelines in constituting the NRP; however, there is no detailed description of the procedure used to choose panelists from about 300 nominees (Camilli, Vargas, & Yurecko, 2003, pg 36 of web document).

Camilli et al. are correct in stating that detailed information about the selection process is not provided in the report.  Additional information about the selection process is detailed in a press release available publicly on the NRP website.  The three hundred nominations were received from "individuals and organizations interested in addressing research-based mechanisms to teach children to read" (NICHD Press Release, 1998). [2]   The press release goes on to state,

In order to obtain as broad a representation of nominees as possible, requests for nominations were sent to ED and NICHD scientists involved in reading research and to major reading and other scientific organizations and associations.  The request for nominations was also sent to subscribers of the major electronic mailing lists that serve the reading community (NICHD Press Release, 1998).

The NICHD and Secretary of Education worked collaboratively to identify biases and conflicts of interest among the nominees before making their final selections.  They made their final decisions based upon the Congressional charge, the availability of potential members (some initial selections turned down the invitation due to scheduling demands), and the strength of the recommendation for the nominees

Panel Member Qualifications

Shanahan (2004) provides an excellent summary of the various criticisms of the qualifications of the panel members (See, for example, Yatvin, 2000 and 2002; Garan, 2002; Allington, 2002; Toll, 2001).  In her minority report, Yatvin states,

As a body made up of mostly university professors, however, its members were not qualified to be the sole judges of the "readiness for implementation in the classroom" of their findings or whether the findings could be "used immediately by parents, teachers, and other educational audiences" (Yatvin, 2000, pg 2).

Shanahan (2004, pg. 250-51) clarifies that the purpose of the panel, set forth by the Congressional charge, was to analyze research evidence.  This task requires extensive knowledge of and training in research methodology and analysis, rather than solely classroom teaching experience.  Therefore it is entirely appropriate and necessary that several of the panelists were recognized scientists in reading research with this capability. 

It is important to also note that the panelists were not the "sole judges" of this research endeavor.  Throughout the process, the Panel held a series of five regional meetings and spoke directly with those working in classrooms.  The Panel also met with more than a dozen leading education organizations to learn how the organizations could facilitate implementation of the findings.  Each of the Panel's meetings was open to the public and devoted significant portions of time to public comment and discussion. [3]

Yatvin (2002) and Camilli et al. (2003) have asserted that only classroom teachers, or those with classroom teaching experience, would have been capable of fulfilling the Congressional charge to report on "the readiness for application in the classroom of the results of this research" (Senate Report 105-58, 1997).  Camilli et al. (2003, pg. 11 of web document) state, "Yatvin also observed that only one panel member (Yatvin) had teaching experience, and thus the NRP had little expertise for the purpose of linking research findings to practice."  Not only is this completely incorrect, it is not even an accurate representation of what Yatvin wrote in her 2002 Phi Delta Kappan article.  However, neither claim is correct.  In reality, at least five of the panelists had K-12 teaching experience and several others had experience as teacher educators in schools of education (see list below).

The Panel was well qualified to report on the readiness for application by determining if the studied interventions had worked in different types of settings.  It was clearly beyond the scope of the Panel's charge to provide a comprehensive assessment of the nation's district, school, and classroom level readiness for implementation of specific intervention practices.  Furthermore, even if the entire Panel had been comprised of classroom teachers, it would be difficult to imagine that only fifteen professionals would have the capacity or knowledge to represent all classrooms, schools, and districts in the nation in order to make specific implementation recommendations.  This argument is based on an incorrect interpretation of the Congressional charge.  The Panel was tasked by Congress to identify practices through research that could be used with a variety of students in a variety of contexts and therefore had the potential for implementation and utility in typical American classrooms.

Biographies of each panel member are available on the NRP website. [4]   Each member of the panel had experience in at least one of the areas specifically listed in the Congressional charge: to "include leading scientists in reading research, representatives of colleges of education, reading teachers, educational administrators, and parents" (Senate Report 105-58, 1997).  A listing of the members and which Congressionally mandated area each represented follows:

· Gloria Correro - Representative of a college of education, 3 years teaching experience in elementary and junior high, 30 years experience as a teacher educator

· Linnea Ehri - Internationally recognized scientist for her research on early reading development and instruction

· Gwenette Ferguson - 22 years teaching experience in public middle school reading and language arts, 16 years (of the 22) experience as chair of school Language Arts Department, currently working as a middle school principal

· Norma Garza - Parent of three children, one of whom struggled with reading, founder and chair of the Brownsville Reads Task Force, Brownsville, Texas

· Michael Kamil - Leading scientist in reading research, representative of a college of education

· Donald Langenberg - Educational administrator, former director of National Science Foundation

· Cora Bagley Marrett - Educational administrator , led Directorate for Social, Behavioral and Economic Sciences at the National Science Foundation

· S. Jay Samuels - Leading scientist in reading research; 12 years teaching experience in public elementary schools, 38 years as a teacher educator of over 10,000 pre-service teachers

· Timothy Shanahan - Coordinator of Graduate Programs in Reading, Writing, and Literacy; teacher educator and leading scientist in reading research; 5 years experience in public elementary schools including classroom teaching and Title 1 reading teaching; 24 years as a teacher educator and reading coach

· Sally Shaywitz - Leading scientist in reading research, specifically in reading development and reading disorders

· Thomas Trabasso - Leading scientist in reading research, specifically in comprehension

· Joanna Williams - Leading scientist in reading research, specifically on linguistic, cognitive, and perceptual bases of reading development and disorders

· Dale Willows - Registered school psychologist specializing in reading disabilities, worked for 10 years with school districts and 20 years providing services to K-6 children in a clinic setting, pre-service teacher educator

· Joanne Yatvin (dissenting member) - Educational administrator; multiple years classroom teaching experience (41 years including both classroom teaching and educational administration)

Panel Member Knowledge with Respect to Meta-Analysis

Camilli et al. (2003) comment on the balance of expertise and knowledge on the Panel.  They state,

An appropriate mix of talent may facilitate a knowledge base that furthers dissemination of research findings and improves the design of new research studies.  In this regard, the NRP would have benefited by formal inclusion of one or more methodologists (Note 29).  Alternatively, the research would have benefited from an officially appointed group of expert methodologists charged with translating the NRP's oversight into technically rigorous guidelines for design as well as data collection and analysis (Camilli et al., 2003, pg 36 of web document).

Camilli et al. chose to include information that two expert consultants in methodology were introduced to the Panel in January 1999 and were made available to the NRP members on an as needed basis in an endnote (Note 29), rather than in the text of his study.   The information is, in fact, correct:  Harris Cooper and David Francis served as the expert methodologists advising the NRP in finalizing its research methodology, in the capacity that Camilli et al. state, as an alternative to formal inclusion as a Panel member. [5]

It is interesting to note that in Camilli et al.'s discussion of the principles and methods of meta-analysis, he paraphrases Cooper and Hedges (1994) extensively in an effort to describe the appropriate meta-analysis process (Camilli et al, 2003, pg 7 of web document). [6]   Since Harris Cooper served as one of the Panel's expert methodologists, one would be safe in thinking that he was able to interpret the principles and methods of meta-analysis that he co-authored for the NRP in such a way that the analyses were performed correctly.  It seems, however, that Camilli et al. disagree with that assumption.

Conflicts of Interest

Garan (2002) asserted a wildly false charge that the members of the NRP had vested financial interests in the outcome of the report.  Garan also mentions "many connections" between the panelists and the administration of George W. Bush (Garan, 2002, pg. 76, 80).  As evidence, she puts forth a convoluted web of tenuous connections between individuals who were not members of the NRP panel and the Bush administration.  This "evidence" does not lend credibility to Garan's argument, especially since the people she names were not even on the Panel, and instead unravels into an elaborate game of "six points of separation" (Garan, 2002, pgs. 76-80).

First, the panelists were not permitted to have financial interests in any commercial reading programs and were required to submit financial disclosure statements before being appointed (Shanahan, 2004, pg 252).  Conflicts of interest and financial disclosure were items included in the panel member questionnaire sent to all those who were nominated.  Any responses containing conflicts of interest or financial interest in commercial reading programs were disqualified.  Second, a point that several critics seem to overlook, the NRP was constituted during President Clinton's administration and completed its work a full year before George W. Bush was inaugurated in January 2001. [7]   Therefore, individual panel member's ties to President Bush's administration, whether real or perceived, are hardly relevant.

It is interesting to note that while the NRP panelists were investigated to ensure that they had no financial interest in commercial reading programs prior to their appointment and were unsalaried volunteers in what was an extensive amount of work entailing a great deal of travel away from home, some of those who continue to speak out against the NRP are, in fact, benefiting financially from their critiques of the report.  Several of the frequently cited critics (see, for example, Garan, 2002, 2004; Allington, 2002; and Coles, 2003) have published for-profit books denouncing the NRP report.  One might say that their criticism has turned into personal financial gain in this respect.

Panel Member Roles

There seems to be confusion among critics, including several writing on email lists that serve the reading community , about the distinction between Federal government employees and NICHD-funded researchers and also between NICHD-funded researchers and the panelists (See, for example, Garan, 2002).  As per the Congressional charge that the members of the panel "not [be] officers or employees of the Federal Government" (Senate Report 105-58, 1997) all of the panelists were unsalaried volunteers not employed by NICHD or the Department of Education for their service.  In accordance with policies and procedures governing government panels and committees, NICHD did provide the Panel members with staff support (a secretary, a research assistant, and an executive director), funding for panelists to hire a limited number of research assistants at their places of work, travel expenses and accommodations, a per diem for food, and an honorarium for NRP meetings. [8]

To further clarify the terms frequently confused, we must address the role of NICHD-funded researchers.  NICHD-funded researchers are not employees of the Federal government and receive grants competitively, based on the quality of the proposed research as determined by independent scientific peer review panels under a rigorous and public set of rules. [9]   Any researcher can submit a proposal to NICHD and receive funding if the quality is sufficient, as judged by a panel of their peers.  These researchers are not employed by NICHD and the Institute has no direct control over their work or what they say about it.  NICHD grantees are required to provide progress reports and cite the NIH grant number in published work derived from the grant-supported study.  Furthermore, only two of the original fifteen members of the NRP were actually NICHD-funded researchers, Sally Shaywitz and Thomas Trabasso (Trabasso worked on but was not the principal investigator for an NICHD-funded grant).

Methodology

Study Selection

The procedures used by the NRP to select studies for inclusion in the report are clearly detailed in both the Executive Summary (pg. 27 - 33) and the Report of the Subgroups (pg. 1-5 - 1-11).  The panelists were charged by Congress to address "the effectiveness of various approaches to teaching children to read."  To answer this charge, the Panel completed a search of the PSYCInfo and ERIC databases for studies with an experimental or quasi-experimental design on the selected topics.  The NRP limited their review to studies that actually tried out an instructional practice, procedure, or technique in real-world classroom settings - those employing an experimental and quasi-experimental design.  The criteria used by the Panel "for evaluating research literature are widely accepted by scientists in disciplines involved in medical, behavioral, and social research" (NICHD, 2000, pg. 1-5 & 1-6).  Furthermore,

The application of these criteria increased the probability that objective, rigorous standards were used and that therefore the information obtained from the studies would contribute to the validity of any conclusions drawn (NICHD, 2000, pg. 1-6).

1. [A]ny study selected had to focus directly on children's reading development from preschool through grade 12 (NICHD, 2000, pg. 1-5).

2. The study had to be published in English in a refereed journal (NICHD, 2000, pg. 1-5).

3. Study participants must be carefully described (age; demographics; cognitive, academic and behavioral characteristics) (NICHD, 2000, pg. 1-6).

4. Study interventions must be described in sufficient detail to allow for replicability, including how long the interventions lasted and how long the effects lasted (NICHD, 2000, pg. 1-6).

5. Study methods must allow judgments about how instruction fidelity was ensured (NICHD, 2000, pg. 1-6).

6. Studies must include a full description of outcome measures (NICHD, 2000, pg. 1-6).

Qualitative Research

Several critics have misunderstood the Panel's decision to not include qualitative research in its analysis (Coles, 2001; Cunningham, 2001; Edmondson & Shannon, 2002; Garan, 2002; Newkirk, 2002; Pressley, 2002; Purcell-Gates, 2000).  The decision to limit the meta-analysis to experimental and quasi-experimental research was driven by the type of questions asked of the Panel in the Congressional charge, not by a view that one type of research was "better" than the other.  In a 2003 article in The Reading Teacher, Shanahan states,

Descriptive [qualitative] research cannot establish whether some method, program or material works or not, but it can provide a rich picture of who is using it and what the context of use is like; this is valuable information for researchers as the basis of future research studies and for practitioners to help them enact a particular practice successfully (Shanahan, 2003, pg. 651).

It was not that we did not see the value of qualitative research, but such research could not possibly provide a definitive answer to the questions that Congress raised.  They wanted to know what works and research-tested ways schools could improve reading ability.  Accordingly, we limited our scope to studies that actually tried out the procedure or technique in classrooms under well-described circumstances with appropriate comparison groups (Shanahan, 2003, pg. 652).

Consideration of the Inclusion of Qualitative Research

The Panel did seriously deliberate whether or not to include non-experimental research in its report.  The report states, "The Panel felt that it was important to use a wide range of research but that that research be used in accordance with the purposes and limitations of the various research types" (NICHD, 2000, pg. 1-7).  Some members of the public urged the Panel to include qualitative studies in the regional meetings or during public comment periods of the Panel meetings.  The Panel, and in particular Shanahan, made a sincere and concerted effort to search the extant qualitative research for high quality studies addressing the topics being investigated by the Panel.  Shanahan provides a detailed account of the Panel's attempt to include correlational and case studies to "temper and extend the conclusions drawn" from the experimental studies (Shanahan, 2003, pg 652).

Shanahan concludes that despite his attempts to contact several sources, such as the directors of Federal research centers and the editors of major reading research journals, he "couldn't find [rigorous] qualitative studies that addressed the topics on which [the Panel was] issuing findings.  Because of this [he] dropped the idea of trying to include qualitative studies in [the Panel's] analysis" (2004, pg. 252).  The Panel could not include or review that which does not exist.  Each section on the topic areas covered by the Panel includes questions for further research, some of which might be best answered by qualitative research methods.

Since the National Reading Panel concluded its work, new panels have been convened to review aspects of literacy and reading instruction that were beyond the scope of the NRP's Congressional charge (Senate Report 105-58, 1997).  Two of the populations that were not specifically addressed in the NRP were infants, toddlers, and preschoolers and English Language Learners.  Two new panels address these specific populations.

These panels, unlike the NRP, were not asked solely what works questions.  They were asked to give an overview of what we know in addition to synthesizing the state of the research with respect to what works as demonstrated by converging experimental and quasi-experimental studies.  This broader charge enables the new panels to include qualitative studies in their syntheses to address questions that are appropriately answered with these types of methodology.

See the Topics Covered by the NRP section for more detailed information about the work of these two panels.

Selection Criteria: Peer-Reviewed Journals

One of the criteria set forth in the NRP methodology for selecting studies for the meta-analysis was that the study had to be published in a peer-reviewed or refereed journal (NICHD, 2000, pg. 1-5).  A peer-reviewed journal is one in which each article has been analyzed by people with expertise and credentials in the field of study covered in the article before it is accepted for publishing.  Some journals employ a double-blind peer review process, in which neither the reviewers nor the author know each other's identities.  In an online resource for librarians and archivists, the importance of peer-reviewed journals is described:

No researcher can claim to present "the unvarnished Truth" or "the definitive study."  Other scholars must corroborate or refute any work presented.  The vetting process takes a relatively long time; perhaps up to a year.  Some people point out that, in very small fields, it is likely that reviewers may guess the identities of authors when they examine the manuscripts.  Sometimes, if a scholar advances an unpopular or a truly revolutionary point of view, there may be considerable resistance within the peer group.  Nevertheless, peer review is the best mechanism we have to attain scholarly objectivity and to guard against scientific fraud (Brown-Syed, 2003). [10]

Fletcher and Francis also discuss disclosure of research as one of the fundamental principles of scientific research.  They state,

Another important principal of scientific inquiry is that the findings of each study would be scrutinized and evaluated by the scientific community through presentations and papers, the latter typically critiqued by groups of scientists, representing peer review (Fletcher & Francis, 2004, pg. 67).

The two primary purposes of peer review are to "ensure that the highest quality research is identified" and to "help develop a field through systematic and public review of research findings" (Fletcher & Francis, 2004, pg. 67).  Stated simply by Lyon & Chhabra (2004), "Peer review is a minimal benchmark for trustworthiness."  The authors also draw the distinction between original research published in peer reviewed journals and editorials or opinion pieces published by non-refereed periodicals, such as Educational Leadership or Phi Delta Kappan .  While the latter "provide valuable information, these periodicals are not designed to evaluate and publish the results of original scientific studies" (Lyon & Chhabra, 2004, pg. 4 of web version).

To ensure the highest quality of standards and because the Congressional charge required the Panel to answer solely what works questions, all of the studies included in the NRP meta-analyses were from peer-reviewed journals or publications.  In addition, some of the findings from the NRP report have been published in peer-reviewed journals (See, for example, Ehri et al., 2001).

Data Analysis

The Panel followed accepted standards and guidelines for conducting meta-analysis in the social sciences in accordance with Cooper and Hedges (1994).  In fact, Harris Cooper served as one of the two methodologists guiding the NRP's meta-analysis.  There have been a handful of critics voicing their opinions in opposition to the NRP's analytic work.  However, only two groups have actually replicated a piece of the analysis to see if a particular criticism was founded. [11]   One of these is a serious and rigorous replication of the NRP's meta-analysis of phonics instruction by Camilli et al. (2003).  This replication also cites Cooper and Hedges (1994) as a recognized and respected framework for meta-analysis in the social sciences (Camilli et al., 2003, pgs. 6-7 of web document).

Meta-analysis changes and interpretation .  Camilli et al. did not follow the exact same procedures in the meta-analysis that the NRP used.  First, they chose to drop one study originally included and add three that were not in the NRP meta-analysis (Camilli et al., 2003, pg. 3 of web document).  Second, the authors also coded some of the treatment and control groups differently, in an attempt to capture a more nuanced categorization of the degree of phonics instruction, unlike the NRP (Camilli et al., 2003, pg. 9 of web document).  The NRP categorized any program that included systematic phonics as the treatment and any reading program that did not as the control.  Third, Camilli et al. (2003) also added five dependent variables (language, phonemic awareness, alphabetic knowledge, vocabulary, and writing) and treated tutoring as a main effect.  Despite these changes "[they] estimated a smaller though still substantial effect ( d = 0.24) for systematic phonics" (Camilli et al., 2003, pg. 2 of web document).

Several of these changes to the NRP analysis require some discussion.  First, the effect size that the NRP reports for systematic phonics instruction is d = 0.41 (NICHD, 2000, pg. 2-112).  The effect size reported by Camilli et al., as cited in the press and on listserves serving the reading community, was d = 0.24.  In fact, these are estimates on two different parameters and are therefore not directly comparable.  The reason these estimates are measuring different things is because of the change Camilli et al. made in coding the treatment and control groups.  Camilli et al.'s estimate compares "more" phonics instruction with "less" phonics instruction using the same studies that were included in the NRP meta-analysis. [12]   However, the NRP estimate compares treatments with systematic phonics instruction, as defined in the report (NICHD, 2000, pg. 2-99) to instruction without systematic phonics instruction.  Camilli et al. did estimate the same parameter as NPR in their analysis and came to the result of an effect size of d = 0.514 (Camilli et al., 2003, pg. 26 of web document, Table 5).

Another change Camilli et al. made in their analysis was to code "systematic language programs" as a treatment along with phonics "unless there was not another control group available" (2003, pg. 14 of web document).  Not only did Camilli et al. fail to adequately define what they mean by systematic language programs (also termed language-based activities or simply language in Camilli et al.), this recoding introduces potential selection bias with respect to the studies included.

The NRP meta-analysis that Camilli et al. reanalyzed was not designed to test an effect of systematic language programs.  Recoding the NRP studies for the presence of language-based activities in the interventions does not give an unbiased estimate of the language effect because the studies included in the NRP analysis were selected based on the presence of a phonics instruction.  Therefore, these studies were not representative of the entire scope of available empirical research on using language-based interventions.  The estimate of the language effect in Camilli et al.'s reanalysis must be interpreted with this potential selection bias in mind.

A similar methodological problem occurs with the way Camilli et al. treated the service delivery model of tutoring .  Camilli and Wolfe (2004) state,

.Camilli, Vargas, and Yurecko (2003) reanalyzed the same data, also using meta-analysis, and found that the effect size of programs using systematic phonics was only half as large as that reported by the National Reading Panel [referring to the d = 0.24 vs. d = 0.41].  This effect, moreover, was substantially smaller than the facilitative effect of one-on-one instruction (Camilli & Wolfe, 2004, pg. 26).

The studies included in the NRP meta-analysis, which were the same studies Camilli et al. reanalyzed, [13] were not selected for inclusion of tutoring and are therefore not representative of the universe of tutoring studies available, leading again to the potential for selection bias.  The authors did not look at tutoring, per se, but only at tutoring within phonics studies.  Camilli et al. tried to make tutoring a main effect without including all tutoring studies, such as those that provide instruction in fluency, phonemic awareness, comprehension strategies, etc.  The conclusion that can be fairly made with respect to the Camilli et al. findings for tutoring is that group or class systematic phonics instruction was not as effective as individually taught phonics instruction.  One cannot, however, conclude that tutoring is generally better and more effective than phonics instruction since they are confounded for these studies.

In summary, Camilli et al.'s findings support the conclusion that phonics is effective and it would be even more effective if it were offered to students through tutoring.  This is not the same thing as saying tutoring is more effective than phonics.  The tutoring effect was always estimated in the presence of phonics instruction.  In the brief summary of the results appearing in Educational Leadership and quoted above, Camilli and Wolfe's statement invites the reader to assume independent estimates of the phonics and tutoring effects are being compared, when in fact, this is not the case.

At times, the language used to describe the regression analysis is uneven.  In the abstract, Camilli et al. write, "Systematic phonics instruction when combined with language activities and individualized tutoring may triple the effects of phonics alone" (2003, pg. 2 of web document).  Their results do indeed suggest this finding.  However, in the discussion of the reanalysis, Camilli et al. wrote,

.we would conclude that the advantage of systematic phon ics instruction over some phonics instruction is significant, but cannot be prioritized over other influences on reading skills.  The regression model suggests, furthermore, that the results of phonics, tutoring and language activities are additive (2003, pg. 33 of web document).

The use of the word prioritized is confusing here.  The effects may be additive but they are not independent, at least not in the studies that Camilli et al. are interpreting.  Phonics instruction is always present, so in this sense they can be prioritized here.  The tutoring and language effects are predicated on the presence of phonics instruction.

Interpretation of results .  Even though Camilli et al. claimed to take an "agnostic stance with respect to reading theory and the previous NRP results" (2003, pg. 27 of web document), the authors did not present an unbiased interpretation of their findings.  For example, Camilli et al. interpret their effect size ( d = 0.24) as "substantially smaller" than the NRP effect size (d = 0.41) (Camilli et al, 2003, pg. 3 of web document).  Later in the paper, the authors say that their estimate of the difference between systematic and less systematic phonics ( d = 0.27) is "30% smaller" than the NRP estimate (Camilli et al., 2003, pg. 26 of web document).  Camilli & Wolfe describe the Camilli et al. effect as "only half as large" as the NRP effect (Camilli & Wolfe, 2004, pg. 26).  They present one side of the interpretation of their findings, albeit inconsistently.

However, the authors could have also stated that the absolute difference between these estimates was between d = 0.10 to 0.17.  By Cohen's (1988) standards, this difference is less than small.  Most importantly, and easily lost sight of in this web of competing interpretations, is the question of how this difference in estimated effect sizes should impact a superintendent's decision about whether to adopt a systematic phonics program in the early grades as a part of an integrated reading curriculum.  The answer is not much .  The difference in findings is not substantial enough for local decision-makers to change course about what type of instruction the research shows to give the most benefit to their young students.

Similar conclusions .  Despite the differences in the way the authors conducted the meta-analysis and reported their findings, Camilli et al. reached the same conclusions as the NRP with respect to practice.  Camilli et al. (2003, pg. 37 of web document) cite the following two passages from the NRP report as being very consistent with their own conclusions.

Programs that focus too much on the teaching of letter-sound relations and not enough on putting them to use are unlikely to be very effective.  In implementing systematic phonics instruction, educators must keep the end [original emphasis] in mind and insure that children understand the purpose of learning letter-sounds and are able to apply their skills in their daily reading and writing activities (NICHD, 2000, pg. 2-96).

Finally, it is important to emphasize that systematic phonics instruction should be integrated with other reading instruction to create a balanced reading program.  Phonics instruction is never a total reading program (NICHD, 2000, pg. 2-97).

It is impressive that Camilli et al. used different meta-analytic methods - most notably focusing on different comparison groups and using different effect size weighting schemes - but came to similar conclusions with regard to their major findings.  In particular, based on the findings of the NRP and Camilli et al., both sets of meta-analysts found:

1. The advantage of instructional approaches that include systematic phonics over instructional approaches that include some or no phonics instruction is statistically significant.  However ,

2. Instructional approaches that include systematic phonics instruction show their greatest advantage in the early grades, and

3. Instruction in systematic phonics should be integrated with other types of reading instruction to create a balanced reading program.  Phonics instruction is never a total reading program.

Camilli et al. did a generally impressive job in choosing meta-analytic methods and providing good rationales for their choices.  So did the NRP, generally.  No approach is perfect and we can and should critically appraise all these efforts.  The fact that Camilli et al. and the NRP reviews approached the evidence differently and came to the same broad conclusions is reassuring.

Topics Covered by the NRP

It may come as a shock to some, due to the disproportionate attention the phonics section of the NRP report has received in the media and from the critics, that phonics was one subtopic of one of the five major topics covered in the publication.  For example, Camilli and Wolfe (2004, pg. 27) state incorrectly that the effect of systematic phonics "was the focus of the National Reading Panel's report."  As stated in the NRP Introduction and Methodology Section (NICHD, 2000, pg. 1-2),

Following the regional hearings, the Panel considered, discussed, and debated several dozen possible topic areas and then settled on the following topics for intensive study:

· Alphabetics

- Phonemic Awareness Instruction

- Phonics Instruction

· Fluency

· Comprehension

- Vocabulary Instruction

- Text Comprehension Instruction

- Teacher Preparation and Comprehension Strategies Instruction

· Teacher Education and Reading Instruction

· Computer Technology and Reading Instruction

Topic Selection Process

Yatvin first raised a question about the topic selection process in the minority report by stating,

At the first meeting in the spring of 1998, the Panel quickly decided to examine research in three areas: alphabetics, comprehension, and fluency, thereby excluding any inquiry into the fields of language and literature.  After some debate, members agreed to expand their investigations to two other areas: computer-linked instruction and teacher preparation.(Yatvin, 2000, pg. 1).

Several items are not included in her account of the events.  During the first meeting of the Panel in April 1998 the panelists heard from members of the National Academy of Sciences' National Research Council [NRC] Committee on the Prevention of Reading Difficulties in Young Children.  Preventing Reading Difficulties in Young Children (Snow, Burns & Griffin, 1998) was a consensus document that the NRP build upon and expanded in their analysis of research literature.  Initially, the NRP agreed to include the topics that were covered by the NRC committee in order to build on their work: alphabetics, comprehension, and fluency. [14]   Between the first meeting and the fifth meeting in November of 1998, six months later, the Panel continued to deliberate and discuss several topics for inclusion as well as receive input from the public before voting to determine the final decision (Shanahan, 2004, pg. 239).  This information is also documented in the publicly available Panel meeting minutes. [15]

Therefore, the Panel did not "quickly decid[e] to examine research in three areas;" they deliberately and thoughtfully began their inquiry with the findings of the widely respected and accepted NRC report.  Additionally, "after some debate" included numerous additions and subtractions of topics (up to 30 at one point) and five regional meetings where the Panel heard substantial public input, and spanned a period of six months.  The Panel focused on the final areas only after this extensive and informed deliberation, determining that the selected topics best answered the Congressional charge and were within the scope of their task.

Throughout their research, the Panel also considered a set of seven questions created by NICHD in collaboration with the original authors of Senate Report 105-58 (1997). [16]   These seven questions are:

1. What is known about the basic process by which children learn to read?

2. What are the most common instructional approaches in use in the US to teach children to learn to read?  What are the scientific underpinnings for each of these methodologic approaches, and what assessments have been done to validate their underlying scientific rationale?  What conclusions about the scientific basis for these approaches does the Panel draw from these assessments?

3. What assessments have been made of the effectiveness of each of these methodologies in actual use in helping children develop critical reading skills, and what conclusions does the Panel draw from these assessments?

4. Based on answers to the preceding questions, what does the Panel conclude about the readiness for implementation in the classroom of these research results?

5. How are teachers trained to teach children to read, and what do studies show about the effectiveness of this training?  How can this knowledge be applied to improve this training?

6. What practical findings from the Panel can be used immediately by parents, teachers, and other educational audiences to help children learn how to read, and how can conclusions of the Panel be disseminated most effectively?

7. What important gaps remain in our knowledge of how children learn to read, the effectiveness of different instructional methods for teaching reading, and how to improve the preparation of teachers in reading instruction that could be addressed by additional research?

Importance of Topics Not Included in the NRP Report

The issue of topics that were not covered in the NRP report must be addressed, as some have assumed that their omission might be an indication that these topics were not important or that instructional practices related to them were ineffective.  It was beyond the scope, both in time and in Congressional charge, to address every single aspect of reading instruction that is viewed as important.  Given the constraints they faced, the Panel focused on specific areas.  However, this narrowing of focus should not be interpreted as an indication of the lack of importance of other topics.  As stated in the NRP report:

It should be made clear that the Panel did not consider these questions and the instructional issues that they represent to be the only [emphasis in original] topics of importance in learning to read.  The Panel's silence on other topics should not be interpreted as indicating that other topics have no importance or that improvement in those areas would not lead to greater reading achievement.  It was simply the sheer number of studies identified by Panel staff relevant to reading (more than 100,000 published since 1966 and more than 15,000 prior to 1966) that precluded an exhaustive analysis of the research in all areas of potential interest (NICHD, 2000, pg. 1-3).

New Panels .  In order to address some of the topics and populations that were not specifically studied in the NRP report, two new panels are underway.  The National Early Literacy Panel [NELP] was established in 2002 to extend the work of the NRP and focus on literacy for children aged zero to five.  The NELP is an initiative of the Partnership for Reading.  The Partnership for Reading includes the National Institute for Literacy [NIFL], NICHD, the U.S. Department of Education, and the U.S. Department of Health and Human Services.  NELP is being administered by the National Center for Family Literacy and funded through NIFL. [17]

The NELP has been charged to address four research questions:

1. What are the skills and abilities of young children aged birth to five years that predict later reading outcomes?

2. What environments and settings contribute to or inhibit gains in children's skills and abilities that are linked to later reading outcomes?

3. What child characteristics contribute to or inhibit gains in children's skills and abilities that are linked to later reading outcomes?

4. What programs and interventions contribute to or inhibit gains in children's skills and abilities that are linked to later reading outcomes?

The first question will be answered with a meta-analysis to identify variables predictive of later reading achievement and will include mostly correlational studies.  Question four addresses issues of effectiveness (or what works questions) of particular programs and interventions and will include experimental and quasi-experimental research.  Questions three and four will be analyzed as moderators of the interventions and instructional practices.  The Panel is currently working on the analysis to address questions two, three, and four.  It is estimated that the final report will be completed in August 2004.

The NRP analysis included studies with heterogeneous populations of children.  However, they did not focus specifically on English language learners.  This unique population was included to the extent that they are included in typical American classrooms where the research was conducted.  The NRP recognized early in their deliberations that reading instruction and literacy for students for whom English is a second language is a crucial topic that must be studied but realized that it was beyond their scope, time frame, and expertise to focus on it.  The National Literacy Panel on Language Minority Youth [NLP], established in May of 2002, is funded by the Institute of Education Sciences in the U.S. Department of Education through a contract to SRI International and the Center for Applied Linguistics, is addressing this topic.

The NLP is charged with completing a "comprehensive, formal, evidence-based analysis of the experimental, quasi-experimental, qualitative, and psychometric research literature relevant to a set of selected problems judged to be of central importance in developing literacy in language-minority children and youth, including English language learners (ELLs)" (August, 2002, pg. 1 of web based PDF).  In an activity report for the NLP, August further describes the scope of the Panel's research:

The panel identified five domains that need to be addressed in the review synthesis.  Each domain has one or two over-arching research questions as well as sub-questions.  In addressing the research questions, panelists thought it was important to consider and document, to the extent feasible, the student, contextual, linguistic, and cultural variables that impact language and literacy development for ELLs and language minority children and youth.  The panel also defined literacy in terms of the following component skills: word recognition, fluency, vocabulary, comprehension, spelling, writing, metalinguistic knowledge (concepts of print, for example), and orthographic knowledge.  Variables that influence literacy include: phonological awareness, cultural knowledge, background knowledge, metacognitive strategies, concepts of print, and motivation (August, 2002, pg. 13 of web based PDF).

The five domains are the relationship between oracy and literacy, literacy development, context for literacy development, effective instruction, and assessment (August, 2002, pgs, 13-16 of web based PDF).

Phonics Instruction

Even though the NRP report, Teaching Children to Read (NICHD, 2000) clearly included many topics in addition to phonics, the majority of criticism has been directed at this one area.  The phonics instruction findings have been misunderstood and misinterpreted in several ways.

The Issue with Phonics Instruction

For some critics of the NRP findings in the area of systematic phonics instruction, the issue seems to be the belief that reading instruction is a dichotomous choice between phonics instruction and whole language (See, for example, Coles, 2000 or Krashen, 2002).  The NRP report soundly rejects the notion that reading instruction is a forced choice between two distinct strategies and that systematic phonics instruction and "language based activities," a term used often but not well defined by Camilli et al. (2003), are mutually exclusive (NICHD, 2000, Executive Summary, pg. 11; this false dichotomy has also been addressed by Spear-Swerling & Sternberg, 2001, and Fletcher & Lyon, 1998).

Rather, based on the research of what works, the NRP finds that integrated reading programs that include systematic instruction in phonics are effective for improving students' reading ability (NICHD, 2000, pg. 2-112).  Since the NRP findings do not point to exclusively teaching phonics as the only reading instruction, systematic phonics instruction strategies are not promoted as being mutually exclusive with other effective instructional practices.  It is interesting to note that proponents of whole language instruction devoid of systematic phonics instruction ( extreme whole language) do not offer any scientific evidence to support their claim that whole language is a more effective practice than a comprehensive reading program including systematic phonics instruction.

Other critics readily agree that the existing body of reading research supports the importance of phonics knowledge and decoding skills in students' reading development but have disparate views on how those skills should be taught.  For example, Allington writes,

There is no real argument about whether good readers need to develop rapid, accurate decoding proficiencies.  They do.  The questions rage around the issue of how to develop these proficiencies (Garan, 2002, Foreward, pg. x).

Similarly, Camilli and Wolfe write, ".the issue is not whether 'phonics,' or decoding skills, is important.  The issue is how [emphasis in original] to teach decoding skills (2004, pg. 27)."  Here, again, the NRP is clear.  The research shows that an integrated reading program that includes a systematic phonics instruction component is an effective way to improve reading growth in children (NICHD, 2000, pg. 2-112).

Some critics have misconstrued the NRP report as advocating for a phonics- only approach to teaching reading.  For example Camilli et al. write in their analysis,

If the NRP results are taken to mean that effective instruction in reading should focus on phonics to the exclusion of other curricular activities, instructional policies are likely to be misdirected (Camilli et al., 2003, pg. 4 of web document).

It is unclear from where Camilli et al. and others may have construed this idea; there is no such claim or assertion within the text of the actual NRP report.  Camilli and Wolfe also state,

The National Reading Panel's report on reading instruction places too much emphasis on systematic phonics.  Instead, teachers should incorporate phonics in a broader program of differentiated instruction (Camilli & Wolfe, 2004, pg. 26).

Again, it is unclear on what Camilli and Wolfe base this interpretation of the NRP report.  How they determined that the report places too much emphasis on systematic phonics when there were five main topics and five subtopics studied in depth is unknown.  Ironically, the NRP actually does recommend, based on the research findings, that systematic phonics should be incorporated into a broader reading program as Camilli and Wolfe themselves suggest (NICHD, 2000, pg. 2-135).  The report supports the finding that systematic phonics instruction is a necessary, but not sufficient in and of itself, component of comprehensive reading instruction.  There is no single factor, or magic bullet, to improve reading.  The most effective reading instructional practices draw upon a variety of methods.  In fact, the following passages show that, based on the meta-analysis of the research, the Panel repeatedly and unambiguously stressed the importance of integrating systematic and explicit phonics instruction into a comprehensive reading program.

These facts and findings provide converging evidence that explicit, systematic phonics instruction is a valuable and essential part of a successful classroom reading program.  However, there is a need to be cautious in giving a blanket endorsement of all kinds of phonics instruction (NICHD, 2000, Executive Summary, pg. 10).

It is important to recognize that the goals of phonics instruction are to provide children with the key knowledge and skills and to ensure that they know who to apply that knowledge in their reading and writing.  In other words, phonics teaching is a means to an end..Programs that focus too much on the teaching of letter-sound relationships and not enough on putting them to use are unlikely to be very effective (NICHD, 2000, pgs. 2-96 and 2-135).

Thus, it will also be critical to determine objectively the ways in which systematic phonics instruction can be optimally incorporated and integrated in complete and balanced programs of reading instruction (NICHD, 2000, Executive Summary pg. 11).

Teachers must understand that systematic phonic instruction is only one component-albeit a necessary component-of a total reading program; systematic phonics instruction should be integrated with other reading instruction in phonemic awareness, fluency, and comprehension strategies to create a complete reading program.  While most teachers and educational decisionmakers recognize this, there may be a tendency in some classrooms, particularly in first grade, to allow phonics to become the dominant component, not only in the time devoted to it, but also in the significance attached (NICHD, 2000, Executive Summary pg. 11).

It is important not to judge children's reading competence solely on the basis of their phonics skills and not to devalue their interest in books because they cannot decode with complete accuracy.  It is also critical for teachers to understand that systematic phonics instruction can be provided in an entertaining, vibrant, and creative manner (NICHD, 2000, Executive Summary pg. 11).

Phonics skills must be integrated with the development of phonemic awareness, fluency, and text reading comprehension skills (NICHD, 2000, Executive Summary pg. 11).

Finally, it is important to emphasize that systematic phonics instruction should be integrated with other reading instruction to create a balanced reading program.  Phonics instruction is never a total reading program..Phonics should not become the dominant component in a reading program, neither in the amount of time devoted to it nor in the significance attached. It is important to evaluate children's reading competence in many ways, not only by their phonics skills but also by their interest in books and their ability to understand information that is read to them. By emphasizing all of the processes that contribute to growth in reading, teachers will have the best chance of making every child a reader (NICHD, 2000, Report of the Subgroups pgs. 2-97 and 2-136).

It will also be critical to objectively determine the ways in which systematic phonics instruction can be optimally incorporated and integrated in complete and balanced programs of reading instruction.  Part of this effort should be directed at preservice and inservice education to provide teachers with decisionmaking frameworks to guide their selection, integration, and implementation of phonics instruction within a complete reading program (NICHD, 2000, Report of the Subgroups, pg. 2-137).

The Basis of the NRP Report Phonics Findings

The Panel conducted a meta-analysis to "compare the effectiveness of systematic phonics instruction to other forms of instruction lacking in an emphasis on phonics" (NICHD, 2000, pg. 2-103).  Typical non-systematic phonics instruction comparison groups consisted of those taught with whole language, whole word, or basal instruction programs (NICHD, 2000, pg. 2-103).  The NRP reviewed 66 treatment-control group comparisons of systematic phonics instruction derived from 38 studies that met the established methodological criteria (NICHD, 2000, pg. 2-109).  The Panel examined six types of outcomes measuring growth in reading or spelling (NICHD, 2000, pg. 2-110):

· Decoding of real words chosen to contain regular spelling-to-sound relationships

· Reading nonsense words chosen to represent regular spelling-to-sound relationships

· Word identification (in some cases, with irregular spelling-to-sound relationships)

· Spelling, assessed using either developmental stages for younger children or number of words correct

· Comprehension of material read silently or orally

· Oral reading of connected text (accuracy)

Effect sizes, or results, were calculated for each individual outcome as well as averaged across all six outcomes for a composite effect on general reading (NICHD, 2000, pg. 2-110).  The NRP report states that the "overall effect size is interpreted as assessing the impact of phonics instruction on growth in reading" (NICHD, 2000, pg. 2-112).  This composite effect size is reported as reading growth in the report.

NRP Findings on the Effectiveness of Systematic Phonics Instruction

The NRP found that systematic phonics instruction is an essential component of a balanced, comprehensive reading program (NICHD, 2000, pg. 2-96) and that "phonics skills must be integrated with the development of phonemic awareness, fluency, and text reading comprehension skills" (NICHD, 2000, Executive Summary pg. 11).  The NRP did not conclude that phonics instruction should be the only reading instruction to take place.  The report states, "The hallmarks of systematic phonics programs are that children receive explicit, systematic instruction in a set of pre-specified associations between letters and sounds, and they are taught how to use them to read ." [emphasis added] (NICHD, 2000, pg. 2-103).  The Panel also concluded that it is "critical for teachers to understand that systematic phonics instruction can be provided in an entertaining, vibrant, and creative manner" (NICHD, 2000, Executive Summary pg. 11).

Camilli and Wolfe report the NRP's conclusions on systematic phonics instruction in a misleading manner:

The National Reading Panel's most important conclusion is that its findings "provide solid support" for the idea that systematic phonics instruction is more effective than alternatives in teaching children to read [NB: This statement is not cited in the article] (Camilli & Wolfe, 2004, pg. 26).

Camilli and Wolfe mislead readers to believe that the NRP found that systematic phonics instruction was a more effective reading instruction strategy than any other single reading instruction strategy, such as phonemic awareness instruction, fluency instruction, comprehension strategies instruction, or vocabulary instruction.  This would indeed lead to the conclusion that the NRP recommends focusing on systematic phonics instruction to the exclusion of other teaching strategies.  However, their restatement of the NRP findings is incorrect.  This distinction and misrepresentation by Camilli and Wolfe is crucial.  The NRP report actually states,

Findings provided solid support for the conclusion that systematic phonics instruction makes a more significant contribution to children's growth in reading than do alternative programs providing unsystematic or no phonics instruction [emphasis added] (NICHD, 2000, pg. 2-132).

The NRP's comparison of effectiveness does not end with alternatives as Camilli and Wolfe's did.  The distinction is that the NRP meta-analysis found that reading programs that included systematic phonics instruction, among other teaching strategies, provided a more significant contribution to children's reading growth than reading programs that did not include systematic phonics instruction.  This leads a reader logically to the very conclusion that the NRP reported: ". systematic phonics instruction is a valuable and essential part [emphasis added] of a successful classroom reading program" (NICHD, 2000, Executive Summary pg. 10, Report of the Subgroups pgs. 2-96 and 2-135).  It is difficult to entertain and address Camilli and Wolfe's critique of the NRP findings when they report them incorrectly.

Findings in Terms of Effect Sizes

The NRP reported its findings of the meta-analyses in phonics instruction in terms of effect sizes.  Effect sizes are used to describe collective findings across many studies using measures of the same outcome(s) in a meta-analysis.  It is a way of pooling the results of each study and then reporting it in a standardized metric.  The NRP report describes effect sizes as follows:

The statistic used to assess the effectiveness of phonics instruction on children's growth in reading was effect size, which measures how much the mean [average] of the phonics group exceeded the mean [average] of the control group in standard deviation units.  An effect size of 1.0 indicates that the treatment group mean was one standard deviation higher than the control group mean, suggesting a strong effect of training.  An effect size of 0 indicates that treatment and control group means were identical, suggesting that training had no effect (NICHD, 2000, pg. 2-112).

One way to understand the NRP results with respect to systematic phonics instruction would be to say that the effect size of d = 0.41 means that half of the children receiving systematic instruction in phonics score higher than over 65% of the children who do not receive systematic instruction in phonics.  In other words, a child scoring at the 65 th percentile for children who do not receive systematic phonics instruction is only scoring at the 50 th percentile for children who do receive systematic phonics instruction.  Perhaps more telling, a child scoring at the 40 th percentile for children who have not received systematic phonics instruction will only be at the 25 th percentile when compared to children who have received systematic phonics instruction.

Garan (2002, pg. 16) incorrectly states that the "panel decided" what effect sizes would be considered small, moderate, or large, despite the fact that Cohen is clearly cited next to this statement in the report, "An effect size of 0.20 is considered small; a moderate effect size is 0.50; an effect size of 0.80 or above is large" (NICHD, 2000, pg. 2-112).  Cohen (1988) sets forth suggested values for effect sizes, which are widely accepted in statistical analyses of the social and behavioral sciences.  Cohen (1988) further illustrates the magnitude of effect sizes by giving examples relating to the differences in heights of teenage girls in the United States and how perceptible these differences would be by sight.  An effect size of 0.20 is the difference between the heights of the nation's population of 15 and 16 year old girls.  A medium effect size of 0.50 corresponds to the difference between the heights of the population of 14 and 18 year old girls.  A large effect size of 0.80 would be the difference in heights between the population of 13 and 18 year old girls.  Another example of a large effect size, in general terms, is the difference in IQ between a typical college freshman and someone with a PhD.

This rudimentary understanding of small, medium, and large effect sizes provides a coarse guide to our understanding of the magnitude of effect.  However, Glass et al. (1981, pg. 104) point out that the effectiveness of a particular intervention should be interpreted in relation to other educational interventions; ideally those that seek to produce the same effect.  Additionally, Glass et al. note that the practical importance of an effect depends on its relative costs and benefits. Cohen himself acknowledges the problems associated with using the terms small, medium, and large out of context with respect to effect size (Cohen, 1988).

Therefore, it might be helpful to put effect sizes in the context of other studies of educational interventions or phenomena.  For example, an effect size of 1.0 approximately corresponds to the difference on achievement tests in elementary school produced by one year of schooling (Glass et al., 1981, pg. 102).  The well-known Tennessee class-size experiment conducted using random assignment of students and teachers found an effect size of 0.23 on standardized reading assessments and 0.21 on curriculum based reading assessments for students in small classes of 15 to 17 students compared with the performance of students in regular class sizes with or without an aide (Mosteller, 1995, pg.121, Table 2).  In another study, the authors found peer tutoring to increase tutee achievement by an effect size of 0.40 and of tutors by 0.33 (Cohen, Kulik, & Kulik, 1982).  Finally, in a research synthesis of programs for at risk students, Slavin and Madden reported that a cooperative learning reading program for third and fourth graders resulted in effect sizes ranging from 0.12 to 0.54 on various reading outcomes including vocabulary, comprehension, oral reading, and language (Slavin & Madden, 1989, pg. 8).

NRP Findings for Particular Subgroups

Grade.  The Panel found that systematic phonics instruction had a greater impact on beginning readers, specifically those in Kindergarten and first grade (NICHD, 2000, pg. 2-114).  This finding is supported by the meta-analysis replication done by Camilli et al. (2003, pg. 34 of web document).  The effect size of phonics instruction on younger children's reading growth (the overall effect of the six outcome measures described in an earlier question) was d = 0.55 in the NRP meta-analysis, which is considered to be a moderate impact.  This means that the average student who received systematic phonics instruction as a component of his/her overall reading program (treatment group) scored 0.55 standard deviations higher than the average student who did not receive systematic phonics instruction (control group). The effect on reading growth in older children was d = 0.27 (NICHD, 2000, pg. 2-114).  Both effect sizes were statistically significant (NICHD, 2000, pg. 2-159, Table 3), meaning that they did not differ from zero simply due to statistical chance.

Reading ability.  The comparisons in the NRP's analysis included students with a range of reading ability, much like the range we find in a typical American classroom.  Students in the comparisons included typically developing (or normal) students, at-risk students, low-achieving readers, and those who had reading disabilities.  The effect sizes, or results, for each of these groups are as follows (NICHD, 2000, pg. 2-160, Table 3):

    Table 3 Comparisons by Reading Ability

Reading Ability & Grade

Overall Mean Effect Size

On Reading Growth ( d )

At-risk Kindergarteners

0.58

At-risk first graders

0.74

Typically developing first graders

0.48

Typically developing 2 nd through 6 th graders

0.27

Low achieving 2 nd through 6 th graders

0.15 (not significant)

Reading disabled 2 nd through 6 th graders

0.32

     Note :  All effect sizes are statistically significant unless otherwise noted.

Therefore, reading programs that included systematic phonics instruction had a statistically significant positive impact on the reading growth of all groups except low achieving 2 nd through 6 th graders, with the greatest effects shown with beginning at-risk readers.  The Panel hypothesized that the lack of a statistically significant impact for the low achieving group might be the result of difficulties such as lack of fluency or reading comprehension skills, rather than decoding.  The NRP signaled a need for further research on these types of students to determine the best way to "remediate their reading difficulties" (NICHD, 2000, pg. 2-117).

Socio-economic status .  The effect sizes for students from low, middle, and varied SES levels were all statistically significant (NICHD, 2000, pg. 2-160, Table 3).  However, the NRP did note that the greatest impact was for students of low-SES, an effect size of d = 0.66 (NICHD, 2000, pg. 2-118).

Inclusion of English Language Learners in the NRP Analysis

Initially, second language learners were the focus of a subgroup formed by the Panel at their first meeting on April 24, 1998.  However, after an initial screening of second language reading reviews, the members decided that addressing the specific needs of English language learners with respect to reading instruction was beyond the scope of the tasks set forth by the Congressional charge.  The Panel members did not have specific expertise in the area of English language learners and it was understood that Congress was considering forming another Panel in the future whose task would be to specifically focus on this population.

However, the studies included in the NRP analysis included a wide range of heterogeneous populations of students.  English language learners are represented in the studies in so far as they are represented in the general population and therefore the findings are generalizable to a heterogeneous population.

See also the previous section on Importance of Topics Not Included in the NRP Report for more information on the National Literacy Panel for Language Minority Youth [NLP] since convened to review reading research specific to English language learners.

NRP Findings on Systematic Phonics Instruction and Reading Comprehension

The ultimate goal in the education of our students in reading is for them to comprehend what they are reading at a level that enables them to be productive members of society.  As stated in the NRP Report, "Comprehension has come to be viewed as 'the essence of reading' (Durkin, 1993)" (NICHD, 2000, pg. 4-39).

Garan (2002, pg. 14) has argued that the positive results of systematic phonics instruction on reading growth are not meaningful because the effect sizes are not as strong, or not statistically significant, for students on comprehension than on word level decoding outcomes.  In a section of her book where Garan asserts that the Panel defined reading as a set of discrete subskills, she states,

The problem with the panel's approach is obvious to real teachers.  We all know that just because a child can, for example, decode phonetically regular words or even read fluently does not necessarily guarantee comprehension.  The ability to decode phonetically regular words on little lists is a different concept [emphasis in original] than comprehending a text...(Garan, 2002, pg. 14).

Despite the tone of this passage being incredibly condescending ("obvious to real teachers"), Garan provides an excellent demonstration of her lack of knowledge of the widely accepted research base on word recognition and the connection between decoding and comprehension.  For example, as cited in Stanovich (1996, pg. 418), Gough (1984, pg. 255) noted, "Word recognition is the foundation of the reading process."  Stanovich goes on to say,

It would indeed be surprising if such a fundamental conclusion were no longer true..Research continues to indicate that word recognition is the foundational process of reading..It is now generally acknowledged that to emphasize the centrality of word recognition is not to deny that the ultimate purpose of reading is comprehension (Daneman, 1996; Juel, 1996) (Stanovich, 1996, pg. 418).

It has been amply documented that skill in recognizing words is strongly related to the speed of initial reading acquisition (Bertelson, 1986; Biemiller, 1977-1978; Curtis, 1980; Gough & Hillinger, 1980; Juel, this volume; Juel, Griffith, & Gough, 1986; Liberman, 1982; Perfetti, 1985; Rayner & Pollatsek, 1989; Stanovich, 1982, 1985, 1986; Stanovich, Cunningham, & Feeman, 1984).  Additionally, there is evidence that this relationship is causal-that the development of word recognition skill leads to increases in reading comprehension ability (Biemiller, 1970; Blanchard, 1980; Chall, 1989; Herman, 1985; Lesgold, Resnick, & Hammond, 1985; Lomax, 1983; Stanovich, 1985); although the situation is undoubtably characterized by reciprocal causation (Stanovich, 1986) (Stanovich, 1996, pg. 418).

It is difficult to understand why Garan and other critics would find it hard to disagree with the notion that word recognition is a necessary skill for reading comprehension, especially when the evidence is so well documented.  It seems logical that a reader cannot comprehend text without being able to decode (at least a majority of) the words and indeed the research supports this hypothesis.  The other strategies and skills used in gaining comprehension, such as vocabulary, context clues, syntactic structure, and text comprehension strategies cannot be employed if a reader cannot decipher the individual words on a page.  For example, Shankweiler & Fowler point out that the use of context alone is not an adequate strategy (See also Stanovich, 1996, pg. 430-34).

The data suggest that context can supplement the word's spelling in reducing uncertainty, but it cannot substitute for code-based processes of word-identification.  A number of demonstrations make clear that, without knowledge of the code, reliance on context is an inadequate strategy for arriving at a word's identity (See Stanovich, 1986, 2000) (Shankweiler & Fowler, in press, pg. 22 of manuscript).

Like reliance on context, the exclusive use of sight word vocabulary does not result in good comprehension, especially in the later grades.  Shankweiler & Fowler state,

Evidence exists that reliance on memorization alone is not a good strategy in the long run.  Byrne, Freebody, and Gates (1992) found that among second graders, poor decoders with good sight word skills comprehended text better than those good decoders with poor sign word skills.  By fourth grade, however, the tables had turned: those who acquired decoding skills early went on to gain better comprehension skills, surpassing in both decoding and comprehension the children who failed to establish good decoding skills initially. (See also Connelly, Johnston, and Thompson, 2001) (Shankweiler & Fowler, in press, pg. 18 of manuscript).

Stanovich also discusses the importance of the distinction that word recognition is crucial, but not enough to ensure accurate comprehension.  He states, "Efficient word recognition seems to be a necessary but not sufficient condition for good comprehension in adults, just as it is in children" (Stanovich, 1996, pg. 419; see also Shankweiler & Fowler, in press, pg. 16 of manuscript).  Therefore, decoding is a necessary, but not sufficient in and of itself, skill that contributes to the ability to comprehend text.  NRP findings in the area of phonics show that it is an important enough skill that it requires explicit instruction within the context of a comprehensive reading program (NICHD, 2000).  Shankweiler & Fowler explain further,

An often-repeated criticism of code-emphasis [systematic phonics] instruction is that by focusing on phonic patterns without reference to meaning, children will miss the point of written language as a means of communication.  This concern is not entirely unfounded.  Explicit instruction in the code cannot end there.  It needs to be coordinated with attention to the meaning of the words and text under study..code-based reading instruction should not preclude efforts to improve vocabulary, narrative, inference making, and other skills important for comprehension (e.g., Beck, McKeown, Hamilton & Kucan, 1997; Beck, McKeown & Kucan, 2002) (Shankweiler & Fowler, in press, pg. 22-23 of manuscript).

It is useful to think of skill in reading as having two main components: the skill that enables the reader to identify individual words in written form, and the skill that enables the reader to interpret the words and weave them into meaningful patterns that yield comprehension of narrative or other kinds of texts (Gough & Tunmer, 1986)..Difficulties in reading can involve either or both of these separable components.  It makes no sense to ask which is more important for reading, since both are necessary (Shankweiler & Fowler, in press, pg. 23 in manuscript).

Skill in word recognition and reading comprehension, though separable in principle, tend to be well correlated, especially in beginning readers (Shankweiler et al., 1999).  This is readily understandable because comprehension depends on accurate word recognition.  Failure to acquire word recognition skills in the first grade greatly reduces a child's chances of developing serviceable reading comprehension skills in the later grades (Shankweiler & Fowler, in press, pg. 24 in manuscript).

The NRP supports this finding throughout the report in a clear and open manner.  The studies that the Panel analyzed provided converging evidence of the already well-documented knowledge that word recognition is key to reading comprehension. The Panel stressed the importance of providing systematic phonics instruction in conjunction with authentic reading and writing for children.  See, for example, the following passages from the NRP report:

It is important to recognize that the goals of phonics instruction are to provide children with some key knowledge and skills and to insure that they know how to apply this knowledge in their reading and writing.  Phonics teaching is a means to an end..In implementing systematic phonics instruction, educators must keep the end [emphasis in original] in mind and insure that children understand the purpose of learning letter-sounds and are able to apply their skills in their daily reading and writing activities (NICHD, 2000, pg. 2-96).

The aim of phonics instruction is to help children acquire knowledge and use of the alphabetic system to read and spell words.  Phonics was expected to exert its greatest impact on the ability to decode regularly spelled words and nonwords.  Phonics instruction was also expected to exert a large effect when spelling was measured using a developmental spelling scale, which gives credit for letter-sound spellings as well as correct spellings (e.g., Bear et al., 2000; Blachman et al., 1999).  These capabilities all benefit directly from alphabetic knowledge..  Phonics instruction was expected to impact text reading processes.  The effect was expected to be significant but smaller because its influence is indirect (NICHD, 2000, pg. 2-113).

Phonics skills would be expected to show effects on text comprehension to the extent that phonics skills help children read the words in texts.  This is one reason why phonics instruction may have exerted less impact on text comprehension outcomes than on word reading outcomes, because the impact is indirect.  In addition, although phonics programs do give children practice reading connected text, the purpose of this practice is centered on word recognition rather than on comprehending and thinking about the meaning of what is being read (NICHD, 2000, pg. 2-123).

With this understanding of the relationship between decoding skills and comprehension, the meta-analysis conducted by the Panel showed that there was indeed an indirect effect of systematic phonics instruction on comprehension outcomes.  These outcomes were not large for all ages and levels of reading ability, but this was predicted due to phonics instruction being an indirect influence on comprehension.  The report states:

The strongest effects [of systematic phonics instruction] occurred on measures of decoding regularly spelled words ( d = 0.67) and pseudowords ( d = 0.60).  These effects were statistically larger than effects observed on the other measures, which did not differ from each other.  This indicates that phonics instruction was especially effective in teaching children to decode novel words, one of the main goals of phonics (NICHD, 2000, pg. 2-113).

Effect sizes on comprehension measures ( d = 0.27) and oral reading measures ( d = 0.25) were statistically greater than zero, indicating that phonics instruction significantly improved children's text processing skills as well as their word reading skills (NICHD, 2000, pg. 2-113).

Note that the effect sizes reported in the above passages include all of the comparisons (35 of the 66 included) in the meta-analysis that used a separate comprehension measure as an outcome (NICHD, 2000, pg. 2-159).  When the effects are analyzed separately for younger and older readers, we see that the impact of systematic phonics instruction on beginning readers' comprehension is even greater.  Seventeen of the comparisons with comprehension as an outcome measure included kindergarteners and first graders; twelve comparisons included students in 2 nd through 6 th grade (NICHD, 2000, pg. 2-159).  The report states:

Systematic phonics instruction produced significantly greater growth than non-phonics instruction in younger children's reading comprehension ability ( d = 0.51).  However the effects of systematic phonics instruction on text comprehension in readers above 1 st grade were mixed.  Although gains were significant for the subgroup of disabled readers ( d = 0.32), they were not significant for the older group in general ( d = 0.12) (NICHD, 2000, pg. 2-133).

Whether growth in reading comprehension is produced generally in students above 1 st grade is less clear (NICHD, 2000, pg. 2-134).

The finding that word recognition has a greater impact on the reading comprehension ability of beginning readers than on more advanced readers is also corroborated in the literature.  For example, in a longitudinal study conducted by Bryne et al. (1992), the data "support the claim that good word identification is more relevant to reading comprehension in the early grades than general language skills of the sort contributing to listening comprehension" (1992, pg. 148).

NRP Findings on Systematic Phonics Instruction and Spelling

The findings from the NRP meta-analysis of systematic showed that kindergartners and first graders who received systematic phonics instruction in reading programs had greater growth in developmental spelling (spelling phonetically rather than conventionally).  The effect size for these students was d = 0.67, a greater than moderate effect.  The meta-analysis did not show that systematic phonics instruction produced growth in conventional spelling for students in second grade and above (NICHD, 2000, pg. 2-134).

NRP Findings on the Role of Teachers in Systematic Phonics Instruction

The role of the teacher in phonics instruction, and more broadly, reading instruction, is enormous.  Camilli and Wolfe discuss the importance of the teacher's role in making educated and informed decisions about which students need instruction in particular reading strategies at what time.  They state,

Effective teachers must understand the particular needs and situations of their students as well as a range of contextualized, evidence-based practices.  Questions of whether direct instruction in decoding or phonics is necessary are less important than the questions of when, why, how, and to whom teachers should provide such instruction (Camilli & Wolfe, 2004, pg. 28).

The NRP report similarly discusses the importance of teachers being able to make decisions to individualize instruction practices based on the performance and needs of particular students.  For example, in discussing the questions of "Does one size fit all?" the report states,

There will be some children who already know most letter-sound correspondences, some children who can even decode words, and others who have little or no letter knowledge.  Should teachers proceed through the program and ignore these students?  Or should they assess their students' needs and select the types and amounts of phonics suited to those needs?  .the latter is clearly preferable. (NICHD, 2000, pg. 2-136).

An effective teacher must be well educated in the various strategies and instructional practices available to teach phonics systematically as well as phonemic awareness, fluency, vocabulary and comprehension strategies.  Even more important, effective teachers must also be able to assess and recognize when a particular student is lacking in a specific area before they can make decisions about how to address the individualized need.  Nowhere in the NRP report does it state that teachers should provide a one size fits all instructional phonics program to all children regardless of their respective abilities and needs.

Systematic and Explicit Phonics Instruction Does Not Equal "Skill and Drill"

Nowhere in the NRP report findings or recommendations do the authors advocate for so called skill and drill teaching of systematic phonics.  The suggestion is that skill and drill refers to basing all teacher instruction on phonics worksheets and not connecting phonics skills to authentic text.  The report provides the following definition of systematic phonics instruction and its distinction from non-systematic phonics instruction, in which a description of teaching in a skill and drill format is notably absent:

The hallmarks of systematic phonics programs are that children receive explicit, systematic instruction in a set of prespecified associations between letters and sounds, and they are taught how to use them to read..However, phonics programs vary considerably in exactly what children are taught and how they are taught (Adams, 1990; Aukerman, 1981) (NICHD, 2000, pg. 2-103).

The report lists and defines five practices of phonics instruction, all of which can be taught systematically.  They are analogy phonics, analytic phonics, embedded phonics, phonics through spelling, and synthetic phonics (NICHD, 2000, Executive Summary pg. 8).  The degree of variability in the way teachers can teach phonics systematically, presented in the report, demonstrates that there is no one size fits all prescribed way to effectively teach phonics systematically.  Furthermore, the report states, "It is also critical [emphasis added] for teachers to understand that systematic phonics instruction can be provided in an entertaining, vibrant, and creative manner" (NICHD, 2000, Executive Summary pg. 11).  It is puzzling as to where critics have gotten the notion that the NRP report advocates for skill and drill phonics instruction; it is clearly not represented in the actual report.


References

Adams, M.J. (1990). Beginning to read: Thinking and learning about print . Cambridge, MA: MIT Press.

Allington, R.L. (Ed.). (2002).  Big brother and the national reading curriculum: How ideology trumped evidence .  Portsmouth, NH: Heinemann.

Almasi, J.F., Garas, K., & Shanahan, L. (2002, December). Qualitative research and the report of the National Reading Panel: No methodology left behind?   Paper presented at the annual meeting of the National Reading Conference, Miami, FL.

August, D. (2002).  National Literacy Panel on the Development of Literacy Among Language Minority Youth: Update on Activities.  Single online document.  Retrieved on 4/12/04 from http://web.archive.org/web/20061003224949/http://www.cal.org/natl-lit-panel/reports/activityrpt-0802.pdf.

Aukerman, R.C. (1981). The basal reader approach to reading .  New York: John Wiley & Sons.

Bear, D.R., Invernizz, M., Templeton, S., & Johnson, F. (2000). Words their way: Word study for phonics, vocabulary and spelling instruction . Upper Saddle River, NJ: Merrill.

Beck, I.L., McKeown, M.G. & Kucan, L. (2002).  Bringing words to life: Robust vocabulary instruction .  New York: Guilford Press.

Beck, I.L., McKeown, M.G., Hamilton, R.L. & Kucan, L. (1997).  Questioning the author: An approach for enhancing student engagement with text .  Newark, DE: International Reading Association.

Bertelson, P. (1986). The onset of literacy: Liminal remarks.  Cognition, 24 , 1-30.

Biemiller, A. (1970). The development of the use of graphic and contextual information as children learn to read.  Reading Research Quarterly, 6 , 76-96.

Biemiller, A. (1977-1978). Relationships between oral reading rates for letters, words, and simple text in the development of reading achievement.  Reading Research Quarterly, 13 , 223-253.

Blachman, B. (2000). Phonological awareness. In M. Kamil, P. Moesenthal, P. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. 3). Mahwah, NJ: Erlbaum.

Blanchard, J. (1980). Preliminary investigation of transfer between single-word decoding ability and contextual reading comprehension by poor readers in grade six.  Perceptual and Motor Skills, 51 , 1271-1281.

Bogatz, G.A., & Ball, S. (1972).  The impact of Sesame Street on children's first school experiences .  Princeton, NJ: Educational Testing Service.

Boruch, R.F., DeMoya, D., & Snyder, B. (2002).  The importance of randomized field trials in education and related areas.  In F. Mosteller and R. Boruch (Eds.), Evidence matters: Randomized trials in educational research .  Washington, DC: Brookings Institution Press.

Brown-Syed, C. (2003). What is a peer reviewed journal? In valinor.ca : A website for librarians and archivists.  Retrieved on 3/31/04 from http://web.archive.org/web/20061003224949/http://valinor.ca/peer-review.html.

Byrne, B., Freebody, P., & Gates, A. (1992). Longitudinal data on the relations of word-reading strategies to comprehension, reading time, and phonemic awareness.  Reading Research Quarterly, 27 , 140-151.

Camilli, G. Vargas, S., & Yurecko, M. (2003).  Teaching children to read: The fragile link between science and federal policy.  Education Policy Analysis Archives, 11 , 1-52.  Retrieved from http://web.archive.org/web/20061003224949/http://epaa.asu.edu/epaa/v11n15/.

Camilli, G. & Wolfe, P. (2004). Research on reading: A cautionary tale. Education Leadership, 61 (6), 26-29.

Chall, J.S. (1989). Learning to read: The great debate 20 years later.  Phi Delta Kappan, 70 (7), 521-538.

Cohen, J. (1988). Statistical power analysis for the behavior sciences (2 nd ed.). Hillsdale, NJ: Erlbaum.

Cohen, P.A., Kulik, J.A. & Kulik, C.C. (1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19 , 237-248.

Coles, G. (2000).  Misreading reading: The bad science that hurts children .  Portsmouth, NH: Heinemann.

Coles, G. (2001).  Reading taught to the tune of the "scientific" hickory stick.  Phi Delta Kappan, 83 (3), 205-212.

Coles, G. (2003). Reading the naked truth: Literacy, legislation, and lies .  Portsmouth, NH: Heinemann.

Connelly, V., Johnston, R. & Thompson, G.B. (2001). The effects of phonics instruction on the reading comprehension of beginning readers.  Reading and Writing: An Interdisciplinary Journal, 14 , 423-457.

Cooper, H.M. & Hedges, L.V. (Eds.). (1994). The handbook of research synthesis . New York: Russell Sage Foundation.

Cooper, H. & Reach, K. (2004).  What is a meta-analysis and how do we know we can trust it?  In McCardle, P. and Chhabra, V. (Eds.), The voice of evidence in reading research (pp. 103-126).  Baltimore, MD: Brookes Publishing

Cunningham, J.W. (2001).  Review of the National Reading Panel report.  Reading Research Quarterly, 36 , 326-335.

Curtis, M. (1980). Development of components of reading skill.  Journal of Educational Psychology, 72 , 656-669.

Daneman, M. (1996). Individual differences in reading skills.  In Barr, R., Kamil, M.L., Mosenthal, P., & Pearson, P.D. (Eds.), Handbook of reading research (Vol. 2, pg. 512-538). Mahwah, NJ: Lawrence Erlbaum Associates.

Durkin, D. (1993). Teaching them to read (6 th ed.). Boston, MA: Allyn & Bacon.

Edmondson, J. & Shannon, P. (2002). The will of the people. In R.L. Allington (Ed.), Big brother and that national reading curriculum: How ideology trumped evidence (pg. 137-153).  Portsmouth, NH: Heinemann.

Ehri, L., Nunes, S.R., Willows, D.M., Schuster, B.V., Yaghoubzadeh, Z., & Shanahan, T. (2001). Phonemic awareness instruction helps students learn to read:  Evidence from the National Reading Panel's meta-analysis. Reading Research Quarterly, 36 , 250-287.

Feuer, M., Towne, L., & Shavelson, R.J. (2002).  Scientific culture and educational research.  Educational Researcher, 31 (8), 4-14.

Fletcher, J. & Francis, D. (2004). Scientifically based education research: Questions, designs, and methods. In P. McCardle & V. Chhabra (Eds.), The Voice of Evidence in Reading Research (pp. 59-80).  Baltimore, MD: Brookes Publishing.

Fletcher, J. & Lyon, G.R. (1998).  Reading: A research-based approach.  In W.M. Evers (Ed.), What's gone wrong in America's classrooms (pp. 49-90). Stanford, CA: Hoover Institution Press.

Fuchs, L.S., Fuchs, D. & Kazdan, S. (1999). Effects of peer-assisted learning strategies on high school students with serious reading problems.  Remedial & Special Education, 20 (5), 309-318.

Garan, E. (2001a). Beyond the smoke and mirrors: A critique of the National Reading Panel Report on phonics. Phi Delta Kappan, 82 (7), 500-506.

Garan, E. (2001b). More smoking guns: A response to Linnea Ehri and Steven Stahl. Phi Delta Kappan, 83 (1), 21-27.

Garan, E. (2002). Resisting reading mandates: How to triumph with the truth .  Portsmouth, NH: Heinemann.

Garan, E. (2004). In defense of our children: When politics, profit, and education collide . Portsmouth, NH: Heinemann.

Glass, G.V., McGaw, B., & Smith, M.L. (1981). Meta-analysis in social research . Beverly Hills, CA: Sage Publications.

Gough, P.B. (1984). Word recognition. In P.D. Pearson (Ed.), Handbook of Reading Research (Vol. 1, pg. 225-253).  New York: Longman.

Gough, P.B. & Hillinger, M.L. (1980). Learning to read: An unnatural act.  Bulletin of the Orton Society, 30 , 171-176.

Gough, P.B. & Tunmer, W.E. (1986).  Decoding, reading, and reading disability.  RASE Remedial & Special Education, 7 , 6-10.

Herman, P.A. (1985). The effect of repeated readings on reading rate, speech pauses, and word recognition accuracy.  Reading Research Quarterly, 20 , 553-565.

Juel, C. (1996). Beginning reading. In Barr, R., Kamil, M.L., Mosenthal, P., & Pearson, P.D. (Eds.), Handbook of reading research (Vol. 2, pg. 759-788). Mahwah, NJ: Lawrence Erlbaum Associates.

Juel, C., Griffith, P.L., & Gough, P.B. (1986). Acquisition of literacy: A longitudinal study of children in first and second grade.  Journal of Educational Psychology, 80 , 243-255.

Krashen, S. (2001). More smoke and mirrors: A critique of the National Reading Panel Report on fluency. Phi Delta Kappan, 83 (2), 119-123.

Krashen, S. (2002). The NRP comparison of whole language and phonics: Ignoring the crucial variable of phonics. Talking Points, 13 (4), 22-28.

Lesgold, A., Resnick, L., & Hammond, K. (1985). Learning to read: A longitudinal study of word skill development in two curricula.  In G. MacKinnon & T. Waller (Eds.), Reading research: Advances in theory and practice (Vol. 4, pg. 107-138).  London: Academic Press.

Liberman, I.Y. (1982). A language-oriented view of reading and its disabilities.  In H. Mykelbust (Ed.), Progress in learning disabilities (Vol. 5, pg. 81-101). New York: Grune & Stratton.

Lomax, R. (1983). Applying structural modeling to some component processes of reading comprehension development.  Journal of Experimental Education, 52 , 33-40.

Lyon, G.R. & Chhabra, V. (2004).  The science of reading research.  Educational Leadership, 61 (6), 12-17.  Available from http://web.archive.org/web/20061003224949/http://www.ascd.org/publications/ed_lead/200403/lyon.html.

Manzo, K.K. & Hoff, D.J. (2003, February 5). Federal influence over curriculum exhibits growth.  Education Week, 22 (21).

McCardle, P. & Chhabra, V. (Eds.). (2004).  The voice of evidence in reading research .  Baltimore, MD: Brookes Publishing.

Mosteller, F. (1995). The Tennessee study of class size in the early school grades.  The Future of Children, 5 (2), pg. 113-127.

Murray, D.M. (1998). Design and analysis of group randomized trials . New York: Oxford University Press.

National Institute of Child Health and Human Development (NICHD). (2000).  Report of the National Reading Panel. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups (NIH Publication NO. 00-4754). Washington, DC: U.S. Government Printing Office.  Also available on-line: http://web.archive.org/web/20061003224949/http://www.nichd.nih.gov/publications/nrp/report.htm.

National Institute of Child Health and Human Development (NICHD). (2000).  Report of the National Reading Panel. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction (NIH Publication No. 00-4769).  Washington, DC: U.S. Government Printing Office.  Also available online: http://web.archive.org/web/20061003224949/http://www.nichd.nih.gov/publications/nrp/smallbook.htm.

National Institute of Child Health and Human Development (NICHD). (March 27, 1998). New Panel to Assess Readiness of Reading Research for Use In Nation's Classrooms. Press Release available from http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/Press/press_rel_3_27_98.htm.

National Reading Panel Website. (1997). Senate Report 105-58 .  Retrieved 3/31/04 from http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/NRPAbout/officialcharge.htm.

National Research Council. (2002).  Scientific research in education .  R.J. Shavelson & L. Towne (Eds.), Committee on Scientific Principles for Educational Research.  Washington, DC: National Academy Press.

Newkirk, T. (2002, April 24). Reading and the limits of science.  Education Week, 39 .

Orr, L.L. (1999). Social experiments: Evaluating public programs with experimental methods . Thousand Oaks, CA: Sage.

Perfetti, C.A. (1985). Reading ability . New York: Oxford University Press.

Pressley, M. (2002). Reading instruction that works (2 nd ed.). New York: Guilford Press.

Purcell-Gates, V. (2000, July). The role of qualitative and ethnographic research in educational policy.  Reading Online, 4 (1).  Retrieved April 9, 2004 from http://web.archive.org/web/20061003224949/http://www.readingonline.org/articles/purcell-gates/

Raynor, K., & Pollatsek, A. (1989). The psychology of reading .  Englewood Cliffs, NJ: Prentice-Hall.

Shanahan, T. (2003).  Research-based reading instruction: Myths about the National Reading Panel report.  The Reading Teacher, 56 (7), 646-655.

Shanahan, T. (2004).  Critiques of the National Reading Panel report.  In P. McCardle & V. Chhabra (Eds.), The Voice of Evidence in Reading Research (pp. 235-265).  Baltimore, MD: Brookes Publishing.

Shankweiler, D. & Fowler, A. (in press). Questions people ask about the role of phonological processes in learning to read.  Reading and Writing: An Interdisciplinary Journal .

Shankweiler, D., Lundquist, E., Katz, L., Stuebing, K.K., Fletcher, J.M., Brady, S., Fowler, A., Dreyer, L.G., Marchione, K.E., Shaywitz, S.E., & Shaywitz, B.A. (1999).  Comprehension and decoding: Patterns of association in children with reading difficulties. Scientific Studies of Reading, 3 , 69-94.

Slavin, R.E. & Madden, N.A. (1989). What works for students at risk? A research synthesis. Educational Leadership , 46(4), 4-13.

Snow, C.E., Burns, M.S., & Griffin, P. (Eds). (1998). Preventing reading difficulties in young children .  Washington, DC: National Academies Press.

Spear-Swerling, L. & Sternberg, R. (2001).  What science offers teachers of reading.  Learning Disabilities Research and Practice, 16 (1), 51-57.

Stanovich, K.E. (1982). Individual differences in the cognitive processes of reading I: Word decoding.  Journal of Learning Disabilities, 15 , 485-493.

Stanovich, K.E. (1985). Explaining the variance in reading ability in terms of psychological processes: What have we learned?  Annals of Dyslexia, 35 , 67-96.

Stanovich, K.E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy.  Reading Research Quarterly, 21 , 360-407.

Stanovich, K.E. (1996). Word recognition: Changing perspectives. In Barr, R., Kamil, M.L., Mosenthal, P., & Pearson, P.D. (Eds.), Handbook of reading research (Vol. 2, pg. 418-452). Mahwah, NJ: Lawrence Erlbaum Associates.

Stanovich, K.E. (2000). Progress in understanding reading: Scientific foundations and new frontiers .  New York: Guilford.

Stanovich, K.E., Cunningham, A.E., & Feeman, D.J. (1984). Intelligence, cognitive skills, and early reading progress.  Reading Research Quarterly, 19 , 278-303.

Toll, C.A. (2001).  Can teachers and policy makers talk to one another?  The Reading Teacher 55 (4), 318-325.

Yatvin, J. (2000).  Minority view.  In Report of the National Reading Panel: Teaching children to read.  An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups (NIH Publication No. 00-4754, pp. 1-6).  Washington, DC: U.S. Government Printing Office.

Yatvin, J. (2002).  Babes in the woods: The wanderings of the National Reading Panel [Electronic version].  Phi Delta Kappan, 83 (5), 364-369.  Retrieved April 12, 2004 from http://web.archive.org/web/20061003224949/http://www.pdkintl.org/kappan/k0201yat.htm.

[1] Retrieved on 4/19/04 from http://web.archive.org/web/20061003224949/http://www.nagb.org/.

[2] Retrieved on 3/29/04 from http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/Press/press_rel_3_27_98.htm.

[3] Minutes from each of the Panel and regional meetings are publicly available at http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/NRPAbout/Meetings_Archive.htm.

[4] Retrieved 3/31/04 from http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/NRPAbout/Biographies.htm.

[5] This information can be found in the Panel meeting minutes for January 21, 1999 retrieved on 3/31/04 from http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/NRPAbout/Panel_Meetings/01_21_99.htm.

[6] Note that Camilli et al. references Cooper and Hedges (1994) directly in the text on page 7 of the web document, but neglects to include the publication in the list of references at the end of the paper.

[7] Dates of the Panels meetings can be confirmed by viewing the publicly available minutes from each meeting retrieved on 3/31/04 from http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/NRPAbout/Panel_Meetings/01_21_99.htm.

[8] Information about staff support and resource assistant funding can be found in the Panel meeting minutes for April 24, 1998 and October 19, 1998 retrieved on 3/31/04 from http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/NRPAbout/Panel_Meetings/04_24_98.htmand http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/NRPAbout/Panel_Meetings/10_19_98.htmrespectively.

[9] See http://web.archive.org/web/20061003224949/http://www.nichd.nih.gov/funding/peer-review.htmfor additional information on the NIH/NICHD scientific review panel process.  The NIH Peer Review Policy and guidelines can be found at http://web.archive.org/web/20061003224949/http://grants.nih.gov/grants/peer/peer.htm.

[10] This information can be found at the following website retrieved on 3/31/04 from http://web.archive.org/web/20061003224949/http://valinor.ca/peer-review.html.

[11] The other one was a study by Almasi, Garas, and L. Shanahan (2002) where the authors attempted to include qualitative studies, including ethnography and case study, in the analysis of the comprehension section of the NRP report.  The authors identified 12 qualitative studies of reading comprehension and concluded that the majority of the studies were flawed and would not be expected to add valid data to the NRP analysis (Shanahan, 2004, pg. 243).

[12] With the exception of the one dropped and three added (Camilli et al, 2003, pg. 3 of web document).

[13] With the exception of the one dropped and three added (Camilli et al, 2003, pg. 3 of web document).

[14] This information can be found in the Panel meeting minutes for April 24, 1998 retrieved on 3/31/04 from http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/NRPAbout/Panel_Meetings/04_24_98.htm.

[15] All Panel meeting minutes, including regular and regional public meetings, can be found at http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/NRPAbout/Meetings_Archive.htm.

[16] This information can be found in the Panel meeting minutes for July 24, 1998 retrieved on 3/31/04 from http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/NRPAbout/Panel_Meetings/07_24-25_98.htm.  The seven questions are also publicly available at http://web.archive.org/web/20061003224949/http://www.nationalreadingpanel.org/NRPAbout/Charge.htm.

[17] Further information on NELP and the description of research questions can be found at http://web.archive.org/web/20061003224949/http://www.famlit.org/ProgramsandInitiatives/FamilyPartnershipinReading/index.cfm.

 

 

Monday

July 12th, 2004

EducationNews.org -

A Global Leading News Source

On Twitter

Oklahoma state testing interrupted by tech glitches -- again http://t.co/G8qL86bXfK #edtech #edchat #education

16 hours ago

Pennsylvania cyber charter school teachers join union http://t.co/3ejVLWWLvV #edtech #edchat #education

16 hours ago

UK officials investigate validity of 'Muslim takeover plot' in Birmingham schools http://t.co/2zzPQfteye #ukedchat #education #edchat

2 days ago

On Facebook

Subscribe

Enter your email to subscribe to daily Education News!

Hot Topics

Career Index

Plan your career as an educator using our free online datacase of useful information.

View All