Orwell Award Announcement SusanOhanian.Org Home


Commentaries

Examining Data and Power Point Spin

Posted: 2003-12-21

You don't have to live in the California Bay Area to be concerned about what's going on in this article, which may be news and may be strategized spin. Read the article. Read the two letters by very savvy researchers questioning its content.


Here's an article from The San Francisco Chronicle, followed by two incisive letters-to-editor which cut through the puffery and ask the questions the reporter should have asked.

Testament to Testing
Schools close 'achievement gap' by pinpointing trouble spots with frequent assessments


Nanette Asimov, Chronicle Staff Writer
December 18, 2003
?2003 San Francisco Chronicle

One of the second-graders at San Francisco's Treasure Island Elementary School spent much of last year in the principal's office as punishment for fighting and throwing things in his classroom. The boy couldn't read, and his teacher had no idea why.

This year, the boy took a test that revealed his specific problem: He did not know the sounds corresponding to each letter. That meant he was not ready to learn which letters made up each word. At last his teacher knew how to help him.

"He's reading now," said Principal Greg John. "It's incredible. He hasn't turned into an angel, but he's not in my office every day now, either."

John and the teachers at Treasure Island have become converts to the increasingly popular practice of using frequent tests to diagnose children's academic needs -- a practice they believe can help low-achieving students soar in school.

A new study of 32 Bay Area schools suggests they are right. Researchers from the Bay Area School Reform Collaborative looked at achievement levels in two groups of 16 schools (kindergarten through eighth grade) with similar ethnic and low-income populations. In one group, black and Latino students were doing as well or better than their white and Asian American classmates. In the other group, the ethnic groups reflected the well-known "achievement gap": Less than a third of black and Latino students typically score at the national average in reading, while more than two-thirds of white and Asian American students scored at or above the national average.

The researchers found that schools where black and Latino students' test scores were rising did many things differently from the lower-achieving schools. Most notably, teachers diagnosed students' needs a few times each week then changed how they worked with the kids based on what the data revealed.

"In the education system of our dreams, ethnicity, language, culture, gender and family income would not be good predictors of academic achievement, " said Merrill Vargo, executive director of the Bay Area School Reform Collaborative, a group that works with low-scoring schools.

The researchers examined four years of California test data and identified 16 schools where black and Latino children bucked the trend with their success, and 16 where they did not. The researchers then spent a year visiting the schools to learn why.

Besides frequent diagnostic tests, they found that at the higher- achieving schools, teachers were more likely to learn how to analyze data and apply it to teaching. Also, principals usually considered closing the achievement gap a primary goal, more people of color held leadership positions, and school goals were usually clear and focused.

The researchers did not name the less successful schools. The successful ones were: Belle Air in San Bruno; Bancroft and Monroe in San Leandro; Campbell, Monroe and Rosemary in Campbell; Chabot, Roosevelt and Fruitvale in Oakland; Chipman in Alameda; Fredericksen in Dublin; Lincoln, Musick and Milani in Newark; and Patterson and Thornton in Fremont.

At Belle Air and Musick, for example, Latino students started out with lower test scores than their white and Asian American classmates but improved at a greater rate than their classmates between the 1998-99 and 2001-02 school years, the study found.

And at Roosevelt, black students gained nearly six times what was required to meet the school's annual academic targets during the four-year period. Asian American students made about 2.5 times what was required to meet the targets.

Treasure Island Elementary was not part of the study. But Principal John acknowledges that last year his school would not have been one of the success stories because black and Latino students missed meeting academic targets altogether. He said he intends to change that.

"Last year we didn't have a culture that looked at data," he said. "Teachers tried their own approaches to addressing reading, but essentially the problems remained unchanged."

This year, he said, "It's a huge change. I can't think of anything we're doing that's not different."

His kindergartners through fifth-graders are now part of the national "Reading First" program that requires teachers to assess children every two weeks. Results are posted online so John can see exactly what's going on.

Every student in grades six through eight is also tested now in reading through a program called High Point. Kids who read below grade level now spend an hour a day receiving intensive help in just the areas they need.

"This is so much better," John said. "It's changed the tone here. The statement 'My kids can't read' doesn't work anymore. Now it's how can't they read?"

State Assemblywoman Loni Hancock, a Berkeley Democrat, said she is impressed with the study's findings and wants more schools using data.

Ironically, Hancock wrote Assembly Bill 356, now heading toward the governor's desk, which would exempt second-graders from taking the state's annual exam. She favors the frequent in-class assessments, however, because "they can be used right away to focus on what teachers can do to help children. "

If the governor signs her bill, Hancock said, she would like the $2 million that would be saved from not testing second-graders to go for classroom assessments.

"To me, it's really stunning that that kind of ongoing testing really does improve children's learning dramatically," she said. The full report can be found at http://basrc.org
=====

To the editor

From Stephen Krashen, Emeritus Professor, USC

The Chronicle's report on a study of 32 Bay Area schools by Bay Area School Reform Collaborative gives frequent diagnostic testing the credit for test score increases ("Testament to testing," December 18). A look at the Collaborative's website gives a different impression. First, there is no hard data, at least not yet. The website only supplies press releases (the full report, according to the website, is "coming soon.") Second, even from the press releases, it is clear that other changes took place in these schools, changes that clearly affect reading test scores, such as more time devoted to recreational reading. A proper statistical analysis can reveal which factors actually contributed to the improvement. Both the Collaborative and the Chronicle need to wait until a scientific analysis of the data is made before jumping to conclusions.

======
To the Editor:
From: Harold Berlak, Senior Research Fellow,
Applied Research Center, Oakland

The headlines of Nanette Asimov's "Testament to Testing" (December 18 SF Chronicle) assert that a study issued by the Bay Area School Reform Collaborative (BASRC) shows that "schools close 'achievement gap' by pinpointing trouble spots with frequent assessments." Reprinted were two pie charts that purportedly demonstrated that schools that test most frequently, "help students achieve". Most who peruse the headlines and scan the text would accept this conclusion as plausible and valid. The article itself is more careful and makes the far more modest and
plausible claim that those schools where teachers assess students' progress on a regular basis are more likely to meet the goals set for schools by the State for the Academic Performance Index (or API).

When teachers pay close attention to students' progress, they do better on all their school work --including their standardized test
performance. This is well documented and certainly no surprise. But this claim is a long way from the spin that the report proves the merits of high stakes standardized tests. There is in fact is nothing whatsoever in the data presented in the article that warrants the
conclusion that high stakes tests such as those used to compute the API improves student academic performance, or in anyway diminishes the
yawning gap in achievement between the races and between rich and poor.

At best the evidence we have is mixed with strong indications from numerous independent studies including the Harvard Civil Rights Project
that show that a major consequence of using standardized tests as the sole or primary measure of academic performance is to restrict equality
of educational opportunity and widen the race gap.

BASRC is a non-profit advocacy group and service provider funded by the Hewlett and Annenberg foundations both supporters of high stakes
standardized testing for assessing public schools. BASRC's avowed mission according to its own website is to further what it calls "data
driven" school reform. What they mean by "data driven" are state and national policies that favor standardized, high stakes standardized
tests and rating systems such as the API. At very least I would expect the reporter to seek independent comment by experts who do not share
BASRC's commitment to high stakes testing as an instrument of reform.

What is doubly disturbing is that when I checked the BASRC website, I found that the report that the article is based on does not yet exist.
What exists is a 44 slide PowerPoint presentation extolling the merits of "data driven" reform based on a report that is forthcoming. This raises question whether the Chronicle reporter had access to the full report or based her article entirely on the PowerPoint summary and self-serving claims by the foundation staff.

If there is to be any hope of diminishing the achievement gap we must as a society provide all children with access to the human and material resources they need if they are to learn and thrive. And as BASRC suggests, we also need regular assessments and "data". But we do our children and schools a great disservice when the only data we accept as a measure of educational progress and growth are standardized test scores.



More Commentaries


FAIR USE NOTICE
This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.