Orwell Award Announcement SusanOhanian.Org Home

Teacher Quality Widely Diffused, Ratings Indicate

Ohanian Comment: In addition to all the other problems, teacher ratings are distributed on a curve--so there have to be as many rating "low" as rating "high."

Dolores Soffientini letter to New York Times:

After each New York teacher's performance is assessed, I propose that each wear a scarlet letter proclaiming how he or she performed: A for "awful," B for "bad," C for "can't cut it," D for "dreadful" and E for "exceptional."

If this sounds ridiculous, that's because it is. Once these teacher "grades" are made public, you can imagine the havoc that will ensue when parents find out that their child is assigned to one of the less than stellar teachers.

It never ceases to amaze me how people who have not spent a single day working in a school can claim to be such experts in how they should be run.

By Fernanda Santos and Robert Gebeloff

The controversial ratings of roughly 18,000 New York City teachers released on Friday showed that teachers who were most and least successful in improving their students̢۪ test scores could be found all around -- in the poorest corners of the Bronx, like Tremont and Soundview, and in middle-class neighborhoods of Queens, like Bayside and Forest Hills.

They taught in schools in wealthy swaths of Manhattan, but also in immigrant enclaves.

They were in similar proportions in successful and struggling schools, and they were just as likely to have taught the most challenging of students and the most accomplished.

The ratings, known as teacher data reports, covered three school years ending in 2010, and are intended to show how much value individual teachers add by measuring how much their students̢۪ test scores exceeded or fell short of expectations based on demographics and prior performance. Such "value-added assessments" are increasingly being used in teacher-evaluation systems, but they are an imprecise science. For example, the margin of error is so wide that the average confidence interval around each rating spanned 35 percentiles in math and 53 in English, the city said. Some teachers were judged on as few as 10 students.

Education officials cautioned against drawing conclusions from numbers that are meant to be part of a broader equation.

"The purpose of these reports is not to look at any individual score in isolation, ever," Shael Polakow-Suransky, the No. 2 official in the city's Education Department, said Friday. "No principal would ever make a decision on this score alone, and we would never invite anyone -- parents, reporters, principals, teachers --to draw a conclusion based on this score alone."

The ratings, which began as a pilot program four years ago to improve instruction in 140 city schools, have become the most controversial set of statistics released by the Bloomberg administration. They came out after a long legal battle and amid anguish and protest among educators; on Twitter posts, some compared their release to a modern-day witch hunt.

"I believe the teachers will be right in feeling assaulted and compromised here," Merryl H. Tisch, the chancellor of the State Board of Regents, said in an interview. "And I just think, from every perspective, it sets the wrong tone moving forward."

In releasing the reports, New York became only the second city in the country where teachers' names and ratings have been publicized. In 2010, The Los Angeles Times hired a statistician and published its own set of ratings, in spite of fierce opposition from the local teachers' union. Many thousands of people visited the newspaper's Web site to check the rankings, though, and Arne Duncan, the federal education secretary, praised the effort, saying, "Silence is not an option."

Whether or not they are made public, such ratings have been gaining currency, in part because they are favored by the Obama administration̢۪s Race to the Top initiative. New York City principals have made them a part of tenure decisions. Houston gave bonuses based in part on value-added measures, though that program was reorganized. In Washington, poorly rated teachers have lost their jobs.

The ratings released on Friday are more than a year old and are based on test results that have been somewhat discredited, since the state later recalibrated the scoring. Still, they offer a peek at the state̢۪s future evaluation system, which will use value-added measures for at least 20 percent of teachers' evaluations.

In simple terms, value-added models use mathematical formulas to predict how a group of students will do on each year̢۪s tests based on their scores from the previous year, while accounting for factors that include race, gender, income level and other test results. If the students surpass expectations, their teacher is rated well -- "above average" or "high" under New York's models. If they fall short, the teacher is rated "below average" or "low."

What many teachers point out is that the scores cannot account for many other factors: distractions on test day; supportive parents or tutors; allergies; a dog continually barking near the test site. There are also schools where students are taught by more than one teacher, making it hard to discern individual contributions. (The reports released by the city gave the same rankings to those teachers.)

"This data is based on ONE test taken on ONE day when several variables, such as child poverty, quite possibly will affect student performance," Lea Weinstein, a teacher at Middle School 45 in the Bronx, wrote to The New York Times in response to her rating. "Yes, I administered this test that generated this data to my sixth-graders two years ago. I no longer teach sixth grade, and I no longer teach in the same school, or even the same subject. How is this data relevant today?"

In New York, the ratings cover teachers in fourth through eighth grades, because of when state tests are given. They are distributed on a curve, so that for 2009-10, 50 percent of teachers were ranked "average"; 20 percent each "above average" and "below average"; and 5 percent each "high" and "low." Teachers received separate reports for math and English, though in the lower grades they generally teach both. The data released on Friday did not include teachers in charter schools or District 75.

In 2010, The Times and other news organizations requested the records under the Freedom of Information Law. The United Federation of Teachers then sued to block their release. As the case made its way to the state's highest court, one of the judges who ruled on it said imperfection was no reason to hide the data.

The union president, Michael Mulgrew, has cited many mistakes in the reports: one teacher was ranked for a semester when she was on maternity leave; some teachers who taught English were ranked in math. On Friday, education officials said that all teachers who were rated had been asked to verify the accuracy of their reports, that 37 percent of them did and that 3 percent of those teachers' ratings were based on courses they had not actually taught.

"I participated in a pilot program during the 2009-10 school year in which I only taught reading and writing, and another teacher taught my students math the whole year," said Donna Lubniewski, a teacher at Public School 114 in the Bronx. "Since these students were listed in my class, the math scores went under my name. This is why this evaluation should not be used to judge teachers."

Still, the reports, which name teachers and their schools, have the potential to color careers or bolster reputations. And they are full of surprises.

At the Ocean School in Far Rockaway, Queens, where virtually every student is poor enough to qualify for free lunch, none of the 12 teachers in the ratings ranked "low" or "below average." The city gave the school a "C" on its progress report last year. At P.S. 290 on the Upper East Side in Manhattan, which got an "A" on its progress report, the 16 evaluations included one "low" and four "below average" ratings.

Over all, however, the teacher data reports tended to be highly correlated to the schools' grades. Last year, 79 percent of high-performing math teachers worked in "A" or "B" schools, according to the Education Department. But there was no relationship between a school's demographics and its number of high- or low-performing teachers: 26 percent of math teachers serving the poorest of students had high scores, as did 27 percent of teachers of the wealthiest.

Across the years covered by the reports, teachers at the top and bottom of the performance scale tended to stay there. Officials said 521 teachers were rated in the bottom 5 percent for at least two of the five years, while 696 were repeatedly in the top 5 percent. And 68 percent of math teachers in the top or the bottom quarter of the rankings in 2009-10 were in the same category in their multiyear scores.

Parents tend to focus on a school's rating, but the data showed variation in a building. At P.S. 230 in Kensington, Brooklyn, seven fourth-grade English teachers ranged from the 12th to the 99th percentile.

Hundreds of the highest-rated teachers taught students considered to be difficult to educate, like those not proficient in English or with special needs -- perhaps because the model predicted those students would perform poorly.

At P.S. 49, in the Mott Haven section of the Bronx, state test scores for the students in a certain fifth-grade math teacher's class should have been in the 13th percentile citywide, based on their demographics and past performance on the exams. Yet under the teacher's guidance, the students ended up scoring close to the city average. The teacher, in turn, was rated "high."

Reporting was contributed by Tyson Evans, Beth Fertig, Winnie Hu, Maria Newman, Sharon Otterman, Chris Palmer and Anna M. Phillips.

— Fernanda Santos and Robert Gebeloff
New York Times





This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.