Orwell Award Announcement SusanOhanian.Org Home


The "Open Court Effect"?

Susan Notes:
Kudos to the UCLA group for responding so quickly to a study for which there is less than meets the eye in the press release. Their judgment: The major finding of the study, therefore, might be best reported as a small difference between similar programs in similar schools. In no way does this study help us understand whether a mandated, highly prescriptive language arts program compares favorably to long term strategies for fundamentally improving the quality of learning in Los Angeles schools.

Here's more comment: EdSource's Press Release on Open Court


Comments on Similar Students, Different Results: Why Do Some Schools Do Better?
Elementary School Curriculum Program and API: A More Detailed Examination


UCLA Institute for Democracy, Education, and Access (IDEA)

April 2006

This month, EdSource and its research partners Policy Analysis for
California Education (PACE) and the American Institutes for Research
(AIR) released a report on the impact on achievement in reading and
mathematics of curriculum programs in reading and mathematics in
elementary schools. The report's press release cited the main
findings as follows:

"The new analysis found that for English language arts, using the
Open Court curriculum program school-wide did appear to make a
difference in a school's API score. Open Court appeared to be most
effective when it was:

used intensively-i.e., all teachers in the school reported
using Open Court daily;

combined with a coherent, school-wide, standards-based
instructional program; and

combined with the frequent use of student assessment data to
improve instruction."

On first read, one might assume that this study provides evidence
that Open Court is superior to other approaches to language arts
instruction in elementary schools, particularly when schools use it
every day, when all teachers in a school use it the same way, and
when the administrators monitor and evaluate teachers with test
data. We suspect that the study will be cited just this way in
policy discussions about whether or not control over curriculum and
instruction should be decentralized to local schools.

In fact, as we describe more below, the study's findings, as
interesting as they may be, are very limited. Open Court was
compared only to one other, quite similar program, and the
comparison was made only in a narrow band of California schools.
Moreover, within these constrained conditions, the achievement
advantage for Open Court was modest.

A Narrow Set of Programs Compared

Let's look first at the programs that the study compared: Open
Court (publisher SRA/McGraw-Hill) and Houghton Mifflin Reading: A
Legacy of Literacy (publisher Houghton Mifflin). These two programs
are the only two K-8 reading/language arts instructional materials
that have adopted in California; they are also the only approved
materials for use in California in grades K-3 under NCLB's Reading
First program. (Reading First funds are provided only to districts
where either one thousand of their second and third graders or fifty
percent or more of their second and third graders scoring in the
categories "below basic" and "far below basic" on the California
Standards Test.)

Both the Open Court and Houghton Mifflin curricula are commercially
produced basal reading programs. Both are aligned to the California
content standards. Both are prescriptive programs, in that they use
direct instruction methods to teach intensive, systematic, and
explicit phonics. In fact, both programs have "scripted" lessons
for teachers to follow. In sum, the two programs are very much
alike.

Because the two programs are so much alike, the EdSource/PACE/AIR
study's findings say nothing about the effectiveness of prescriptive
curriculum (which would include both Open Court and Houghton
Mifflin) in comparison to project-based, culturally relevant
pedagogy developed collaboratively by teachers.

A Narrow Range of Schools Studied

Second, it is important to consider which schools were included in
the study. The researchers selected a group of 227 elementary
schools that share key student and teacher characteristics. To
identify these schools, they used the state-developed Schools
Characteristic Index (SCI), a composite measure of pupil mobility,
pupil ethnicity, pupil socioeconomic status, percentage of teachers
who are fully credentialed, percentage of teachers who hold
emergency credentials, percentage of pupils who are English language
learners, average class size per grade level, whether the schools
operate multi-track year-round schedule to relieve overcrowding.

EdSource/PACE/AIR only studied schools that fall between the 25th to
35th percentile statewide on Schools Characteristics Index. The
researchers described these schools as schools "where the student
demographic challenge factors are substantial, but not the most
severe." In general, these schools enrolled larger percentages of
EL, Hispanic, and low-income students that are represented in the
state as a whole. For example, 98% percent of the schools in the
sample qualified for Title I funding under NCLB. These are also
schools more likely to have larger proportions of less-than-fully-
qualified teachers. Only 4% had API rankings of seven or better in
the state, meaning that the other 96% of the schools fell below 60%
of the state in student achievement, and most were in the bottom
half of the state. Additionally, 34% of the study schools
participated in Program Improvement-meaning that these Title I
schools had not met the federal Adequate Yearly Progress benchmark
for at least two consecutive years. Because the study schools
represent such a narrow band of California schools, we can not
generalize the findings to the state as a whole.

A Small Difference

Finally, it is important to note that the difference in the impact
on achievement between Open Court and Houghton Mifflin Reading is
actually quite small. As the researchers note, the intensive use of
Open Court was related higher API scores in only 3 out of 5
schools. The major finding of the study, therefore, might be best
reported as a small difference between similar programs in similar
schools. In no way does this study help us understand whether a
mandated, highly prescriptive language arts program compares
favorably to long term strategies for fundamentally improving the
quality of learning in Los Angeles schools. That broader goal
likely demands intensive collaboratively developed by fully
qualified teachers and more rigorous curriculum that calls for
higher order thinking skills and connects to the local context.



— UCLA Institute for Democracy, Education, and Access (IDEA)


http://www.idea.gseis.ucla.edu/


INDEX OF RESEARCH THAT COUNTS


FAIR USE NOTICE
This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.