Analysis of the Texas Reading Tests, Grades 4, 8, and 10, 1995-1998 

Sandra Stotsky, Ed.D.
Harvard Graduate School of Education

November, 1998 

The Purpose of this Report

This report provides a comparison of the difficulty level of the reading selections used in the Texas assessments of reading for grades 4, 8, and 10 from 1995 to 1998. The purpose for the analysis was to determine whether the tests at each grade level were comparable in difficulty from year to year and, if not, how they changed in difficulty. The questions are important. In order for policy makers and other citizens to determine whether there is genuine improvement in students’ reading skills from year to year, the tests that are used need to be comparable in difficulty from year to year. A test that is harder than the previous year’s test will mask improvement. Similarly, a test that is easier than the previous year’s test will mask lack of improvement or decline.

What Was Examined

To address the purpose for this report, I noted (1) the number of questions asked on each test, (2) the types of questions asked, or reading skills assessed, (3) the number of selections on each test, and (4) the number of words on each test. I also determined (5) the genre (or literary nature) of the selections used, and calculated (6) the level of difficulty of each selection using the New Dale-Chall Readability Formula. Further, I counted up (7) the number of different words in all sample passages outside the revised Dale list, a list of 3000 words familiar to most students by grade 4 because they are the most elemental words in the English language.

Why These Features Were Examined

A student’s score on a reading test is heavily influenced by the length of the test, the type of question to which the student responds, and the difficulty level of the selections used on the test. These are features of a test that can be objectively determined. The length of a test is determined by counting up the total number of words in its selections. The types of questions used (or reading skills assessed) are spelled out by test developers.

The difficulty level of a selection can be estimated by determining the commonness of its vocabulary, average sentence length, and the number of sentences in the selection. I used the Dale-Chall Readability Formula because it is one of the best-known and most frequently used readability formulas for estimating or predicting the difficulty of reading materials. First published in 1948, it was acknowledged to be the most popular readability formula in educational circles for several decades. As indicated in Readability Revisited: The New Dale-Chall Readability Formula, its predictions of comprehension difficulty were also reported to be the most valid when compared to independent measures of comprehension difficulty-such as multiple choice and cloze tests, teachers’ and/or students’ judgments of difficulty, and other accepted measures of reading comprehenison-and when compared to other classic formulas.

The formula is based on a semantic factor and a syntactic factor: word familiarity and sentence complexity. The semantic factor accounts for about two thirds of the weight of the formula and is determined by the number of words in sample passages from a text outside a list of 3000 common words known by most grade 4 students. The syntactic factor-sentence complexity-accounts for the other third and is determined by the number of complete sentences in those sample passages. The Dale-Chall Formula does not necessarily estimate the complete conceptual difficulty of a selection by its count of unfamiliar words, nor does it capture the influence of text cohesion and organization on the difficulty of a selection. However, it does have extremely high predictive ability (.92) and, as Chall and Dale suggest, can be viewed as predicting the grade level suitable for independent silent reading, without teachers’ instruction or assistance.

The last measure used, (7) above, is an approximate and partial index of difficulty that I have constructed to show changes in the demands of the reading vocabulary in the tests across one grade level. It is not a count of all words outside the revised Dale list in a selection, just those in the sample passages. The number does not tell us exactly how difficult those words are; some may be very difficult, or uncommon, while some may be relatively easy, or common, despite being outside the Dale list. Nevertheless, a version of a test that has many fewer uncommon words than another version of the test at the same grade level is probably less intellectually demanding.

Constants Across Grades and Tests

The test developers indicate that six types of reading skills were tested through multiple-choice type questions at all grade levels and for all tests from 1995-1998. They deal with students’ ability to (1) determine word meanings, (2) understand facts or details, (3) identify main ideas, (4) understand relationships among ideas and actions in a text, (5) make inferences and generalizations, and (6) distinguish between fact and opinion. At each grade level, these six reading skills were tested across all tests the same number of times. The configuration differed somewhat from grade level to grade level (e.g., a few more vocabulary items were tested at grade 4 than at grade 10). At grade 4, there were 40 items on each test, and at grade 8 and grade 10, there were 48 items per test. Thus, differences in reading difficulty across tests at any one grade level cannot be traced to differences in the types of reading skills tested or in the number of times each skill was tested.

Results of the Grade 4 Analysis

Year                                                       
1995         1996      1997         1998

Total no. of words                                 
2824          2486        2739        1746

Number of selections                                   
    7                  6              6            6

Average no. of words per selection          
   403         414           455            291

Total no. of
unrepeated words
outside Dale list in
sample passages                                           
   119              73            82            62

Average no. of unrepeated words
outside Dale list in sample passages
per selection                                                    
17                   12             14             10

Grade level of selections using
DC formula
2 @ 5-6           4 @ 4          3 @ 4          2 @ 4
3 @ 4              2 @ 3          3 @ 3          4 @ 3
2 @ 3

Findings: Analysis of the grade 4 selections indicates that the 1998 test was much easier than the previous three tests. The number of words students read in 1998 was much lower than the number in previous years. In addition, the total number of unrepeated words outside the Dale list in the sample passages in all 1998 selections was lower than the number in the two previous years, and almost half the number in the 1995 test.

The 1995 test was much more difficult than the tests in succeeding years, even though the total number of words students read in 1996 and 1997 was only a little bit lower. (The number was lower because the 1996 and 1997 tests consisted of six, not seven, selections.) In addition, the total number of unrepeated words outside the Dale list in the sample passages from all selections in the 1996 and 1997 tests was much lower than the number in 1995.

The construction of the tests in terms of the level of reading difficulty of the individual selections differed from year to year. The 1995 test contained selections higher as well as lower in difficulty than grade 4, in addition to a large number of selections right on grade level. The tests in succeeding years became progressively easier. From 1996 to 1998, there were no selections above grade 4 level in difficulty. In 1996, most of them were at grade level; in 1997, half were at grade level and half below grade level; and in 1998, most were below grade level in reading difficulty.

With respect to genre, in 1995 two selections were expository in structure (grade levels 4 and 5-6), two were biographical (grade levels 3 and 4), one was informational (grade level 4), and two were fictional narratives (grade levels 3 and 5-6). In 1996, five of the six selections were fictional narratives (three at grade level 4 and two at 3), and only one was expository in structure (grade level 4). In 1997, all were narrative in structure, four fictional (three were at grade level 3 and one at grade level 4) and two biographical (both were at grade level 4). In large part, the structure of the selections in 1996 and 1997 (mostly narrative) accounts for their lower levels of difficulty. But in 1998, only two selections were fictional narratives (grade levels 3 and 4). The fact that two selections were informational and two expository in structure might at first blush suggest that the 1998 test could have been higher in overall difficulty level than the two previous years. But these informational selections were both at grade level 3, and the expository selections were at grade levels 3 and 4 and thus very easy reading for the end of grade 4. Here is a paragraph from the expository selection on seaweed (grade level 4):

"Seaweed is a plant found in oceans all over the world. It may be very tiny or longer than a football field. Some seaweed looks like long ribbons. Other seaweed is branched or feathery. Scientists group seaweed by color into three main types: green, red, and brown."

Here is part of a paragraph from the expository selection on penguins (grade level 3):

"When the chicks are old enough to feed themselves,they go to the sea while their parents return to land to molt. After the adult penguins shed their old feathers and grow new ones, they go back to the sea. Penguins are more suited to life in the ocean than on land. They sleep in the ocean and even drink the ocean water."

 

Results of the Grade 8 Analysis
Year      1995      1996       1997      1998

Total no. of words
            4261          3791      3884        3120

Number of Selections
             8                6             6             6

Average no.
of words per selection
            533            632         647         520

Total no. of unrepeated words
outside Dale list in sample passages
            301            218          297         161

Average no. of unrepeated words
outside Dale list in sample passages
per selection
             38              36            50           27

Grade level of selections using
DC formula
      2 @ 9-10    1 @ 9-10   1 @ 11-12       2 @ 9-10
      3 @ 7-8      3 @ 7-8     1 @ 9-10         1 @ 7-8
      2 @ 5-6      1 @ 5-6     2 @ 7-8           1 @ 5-6
     1 @ 4          1 @ 3          2 @ 5-6          1 @ 4
     1 @ 3

Findings: Analysis of the grade 8 tests indicates that the 1998 test is easier than those in previous years. The number of words students read in 1998 was much lower than in previous years, even though the number of selections students read in 1998 was the same on the 1996 and 1997 tests. In 1995, 1996, and 1997, over half of the selections were at or above grade level. But in 1998, only half were at or above grade level. Indeed, the number of unrepeated words outside the Dale list from the samples in all selections is much lower in 1998 than in previous years, even though the total number of words per selection in 1998 (520) is not much different from the total number of words per selection in 1995 (533). The 1997 test was harder than the 1995 test, in large part because of the biographical selections used. Indeed, the total number of unrepeated words outside the Dale list for the sample passages in all selections was highest in 1997.

With respect to genre, in 1995, four selections were fictional narratives (one at grade level 4, two at 5-6, and one at 7-8), two were informational (one at 7-8 and one at 9-10), and two were expository in structure (one at 7-8 and one at 9-10). In 1996, two were fictional narratives (one at 3 and one at 5-6), two were expository in structure (one at 7-8 and one at 9-10), one was biographical (7-8), and the last selection involved a comparison of information in two letters-an appropriately challenging selection (7-8). In 1997, there were three biographical selections (7-8, 9-10, 11-12), one informational selection (7-8), one fictional narrative (5-6), and a comparison of information in two letters (5-6). In 1998, there were three informational selections (one at 7-8 and two at 9-10), one expository selection (5-6), and two fictional narratives (3 and 4). The decrease in overall reading difficulty in 1998 can be accounted for in part by two easy fictional narratives and a relatively easy expository selection for that grade level.

Results of the Grade 10 Analysis

Year                1995       1996      1997       1998

Total no. of words
                       6510       4233      3396       4546

Number of Selections
                      9               7              6             7

Average no. of words per selection
                     723          603          566          649

Total no. of unrepeated words outside
Dale list in sample passages
                     443          257          261         349

Average no. of unrepeated words outside
Dale list in sample passages per selection
                     49             37           44             50

Grade level of selections using DC formula
2 @ 11-12       3 @ 9-10     1 @ 11-12        2 @ 11-12
5 @ 9-10         2 @ 7-8        3 @ 7-8           1 @ 9-10
2 @ 5-6          1 @ 5-6         1 @ 5-6            3 @ 7-8
                        1 @ 4           1 @ 4               1 @ 4

Findings: Analysis of the grade 10 tests indicates that the tests became progressively easier in 1996 and 1997. The 1998 test became harder than either of these two tests and is closer in difficulty to that of the 1995 test, although it is still not as difficult as the 1995 test. According to the DC formula, seven of the nine selections in 1995 were at or above grade level in difficulty. In 1996, four of the seven selections were below grade level in difficulty, and in 1997 five of the six were. In 1998, three of the seven selections were at or above grade level in difficulty, two of them much higher.

With respect to genre, the 1995 test contained six expository selection (five at 9-10 and one at 11-12), one comparison of letters (11-12), one informational selection (5-6), and one fictional narrative (5-6). The 1996 test contained one expository selection (7-8), two fictional narratives (4 and 5-6), three biographical selections (one at 7-8 and two at 9-10), and a comparison of letters (9-10). The 1997 test contained two expository selections (7-8 and 11-12), two fictional narratives (4 and 5-6), one informational selection (7-8), and only one letter to analyze (7-8). The 1998 test contained three fictional narratives (one at 4 and two at 7-8), two biographical selections (9-10 and 11-12), and two expository selections (9-10 and 11-12). The 1995 test was more difficult than the others in large part because of the comparison of letters (a good reading exercise) and the large number of grade level expository selections.

Summary and Conclusion: Analysis of the reading tests at all grade levels and for all four years indicates that the tests given from 1995 through 1998 are not comparable in difficulty to each other at any of the grade levels tested. The 1995 tests are longer and more difficult than the 1998 tests at all grade levels. The 1998 tests in grades 4 and 8 are much easier than those in earlier years. The 1998 test in grade 10 is harder than the tests given in 1996 and 1997 although not as difficult as the one given in 1995. If the scores students achieved on the 1998 tests were higher than those achieved by their counterparts on the 1995 tests, the decline in the overall level of reading difficulty of the selections on these tests, as determined by the New Dale-Chall Readability Formula, suggests that there may have been no real improvement in their reading skills. There may have even been a decline. We simply don’t know.

A final question that needs to be addressed is how difficult the 1995 tests were. Were they on grade level? To judge by the results of the New Dale-Chall Readability Formula, the tests used in 1995 at all grade levels were appropriate grade level tests.

In grade 10, there were as many selections below grade 9-10 as above 9-10, but most were at grade 9-10. On the other hand, in 1998, for grade 10, only two selections were above grade level, while four were below it, suggesting that this test is not as demanding as it should be for grade 10.

In grade 8, the 1995 test had two selections above grade 7-8, three selections below grade 7-8, and three selections right at grade level. This test is also right on target. But by 1998, there were three selections below grade level (two considerably below), and only one selection on grade level. Even though there were still two selections above grade 7-8, the pitch of the test is clearly below grade level.

In grade 4, the 1995 test had two selections above grade level, two below, and three right at grade level. By 1998, there were only two selections at grade level; four were below grade level. This test is much too easy for grade 4 students. Academic expectations should be increased over the years, not decreased.

A well-balanced test should have as many selections above as below grade level in reading difficulty to allow evenly for the range of reading ability at each level assessed. But most of its selections should at least be on grade level. It is useful to recall that because the tests are given at the end of the school year, not at the beginning or middle of the school year, many students are closer to the next year’s grade level in terms of reading level than the grade level they are in. Thus, for example, the bulk of the grade 4 test selections could be at grade level 4 and grade level 5-6, using the Dale-Chall formula, and still be at the appropriate level for students at the end of grade 4.

The selections used in recent years have not been chosen with the most appropriate criterion in mind. Clearly, it is desirable to offer a range of genres, to ask students to compare information in two pieces of writing, and to portray various gender roles and social groups. However, the most important criterion for a test of reading is reading level. Selections for future tests should be chosen with an appropriate reading level as the first and chief criterion, as determined chiefly by the demands of their vocabulary. If growth in reading ability is to have real meaning, it must mean growth in the development of a reading vocabulary.