Orwell Award Announcement SusanOhanian.Org Home

Who's teaching L.A.'s kids?

Thank you thank you to Stephen Krashen for immediately sending this letter to the Los Angeles Times. I'm really glad he picked up on the reporters' analyzing teacher lessons. Their comments are pitiful. Another pitiful/tragic thing in all this is that the teachers declared "weak" by the LA Times formula seem to believe the formula: Test scores = Excellence (of both student and teacher).

Everyone--EVERYONE--should send a version of Steve's letter to the LA Times. Here is the form for sending a letter to the editor.

Do it!

letter from Stephen Krashen:

The LA Times ("Who's teaching LA kids?" August 15) is practicing educational research without a license, and is practicing it poorly. The value-added analysis of teacher quality used by the Times is based on test scores. It assumes that the test scores are valid, that teachers are randomly assigned to classes, and that the results for individual teachers are stable. None of these assumptions have been validated.

The analysis also ignores the potent influence of other factors on student achievement, such as poverty, which is typically a stronger predictor of student achievement than teaching quality.

The Times also neglected to follow the usual procedure of sharing work with the academic community before making it public to millions of readers. They also put reporters in the role of expert observers of teachers. What's next, a value-added analysis of brain surgery, with newspaper reporters critiquing surgical procedures?

Ohanian Comment: So now The Los Angeles Times takes it upon itself to declare who's a good teacher and who isn't. Here is "methods paper" supplied by reporters.

"Value Added Measure" has plenty of critics For starters, See "Student Sorting and Bias in Value-Added Estimation: Selection on Observables and Unobservables" by Jesse Rothstein, Education Finance and Policy, Fall 2009. But reporters aren't exactly known for reading studies.

The Los Angeles Times holds VAM up as scientific, reliable, and altogether wonderful in sorting good teachers from bad. And they make teachers' scores public using this measure. The Times claims this is an "analysis, using data largely ignored by LAUSD, looks at which educators help students learn, and which hold them back." The Times notes.

Though controversial among teachers and others, the method has been increasingly embraced by education leaders and policymakers across the country, including the Obama administration.

Of course. Duncan would like this deal which assumes the school puts 25 kids in a box and then pours knowledge into them. Or doesn't.

The late Gerald Bracey summarized the issue in a Huffington Post column:

VAA makes more sense than the current successive-cohorts system for determining AYP. It makes more sense to follow kids over time, although if the goal remains 100% proficiency the whole operation remains nuts.

VAA is circular: it defines effective teachers as those who raise test scores, then uses test score gains to determine who's an effective teacher.

Aside from Sanders, those working in VAA (Henry Braun, Howard Wainer, Dan McCaffrey, Dale Ballou, J. R. Lockwood, Haggai Kupermintz, from all of whom I had quotes) acknowledge that it cannot permit causal inferences about individual teachers. At best, it is a beginning step to identify teachers who might need additional professional development.

Note: According to this LA Times account, John Smith, "the least effective" teacher, got a late start, entering teaching when he was 49.

Prediction: The Los Angeles Times has just increased test prep in the city schools by 500%. . . or 1000%

VTibbets Comment: Your article said."Each student's performance is compared with his or her own in past years, which largely controls for outside influences often blamed for academic failure: poverty, prior learning and other factors." Exactly how does this Value Added method control for outside influences? Do you really think that most of these kids' family lives are simply constant year after year? What about divorce, alcoholism, health concerns, joblessness, and poor nutrition in the family? Don't these affect how a student might perform on a test given one week out of the entire school year? Get a clue. Poor students do, on average, more poorly than rich students. Period.

Michael Paul Goldenberg Comments on EDDRA2 and on Rational Mathematics Education:

I'm not sure what I was doing or where I was the day the LOS ANGELES TIMES went from being a newspaper to being a national leader in evaluating teacher quality. Perhaps it was supposed to be kept secret, like what was discussed and said at the meeting Dick Cheney had with Big Oil executives a decade ago. If so, several TIMES reporters have blown it with a recent article, WHO'S TEACHING LA'S KIDS?

In it, three reporters, Jason Felch, Jason Song and Doug Smith, present ratings of "teacher effectiveness." In particular, they single out one particular fifth grade teacher, John Smith, and claim he is the least effective teacher for his grade level in his school. Mr. Smith's photo is at the beginning of the article that purports to know, based on kids' scores on administrations of a single standardized test, which teachers are helping their students, teaching effectively, and making a positive difference, and, of course, which, like Mr. Smith, are allegedly failing to move their students ahead.

I wrote the following to these reporters today and will be fascinated to see if any of them respond. I know that were I John Smith, I'd be speaking to my attorney and considering lawsuits against several parties, not the least of whom are Jason Felch, Jason Song, and Doug Smith. As I am not, the best I can do is try to point out how wrongheaded, how irresponsible, and how ultimately counterproductive is both their article and the methods they employ to smear the professional integrity of many fine teachers who for any number of reasons may not "measure up." The professional integrity I call into question, however, is that of these reporters, their editors, and others who profit from the publication of this sort of cheap-shot, ignorant journalism.
If you start with the absurd assumption that multiple-guess standardized test scores tell us anything (let alone EVERYTHING) we need to know about teacher effectiveness or student learning of subject matter or all the other things that teachers and schools are about (not all of which are good, but that's another debate entirely), then it follows that the LA TIMES is as qualified as anyone else with no expertise in psychometrics to determine which teachers are "most effective" and which are "least effective."

Further, with the same starting assumption, there's nothing unconscionable about reporters and editors from that noble publication choosing to print a photo of a so-called "ineffective" 5th grade teacher and include the following in the article:
Yet year after year, one fifth-grade class learns far more than the other down the hall. The difference has almost nothing to do with the size of the class, the students or their parents.

It's their teachers.

With Miguel Aguilar, students consistently have made striking gains on state standardized tests, many of them vaulting from the bottom third of students in Los Angeles schools to well above average, according to a Times analysis. John Smith's pupils next door have started out slightly ahead of Aguilar's but by the end of the year have been far behind."
But if the assumption is false, then what the TIMES and its reporters have done is to pillory one 5th grade teacher on the wheel of meaningless test scores. They have, in fact, violated two of the fundamental principles of psychometrics: never use a test designed to measure one thing (e.g., student achievement) to measure something else it was not designed to measure (e.g., teacher effectiveness), and never use a single test score or measurement type to draw definitive conclusions (particularly not in the social sciences). Further, they have violated the notion that correlation (Teacher A's kids scores are higher than Teacher B's scores) with causation (Scores rose primarily BECAUSE of the superior teaching skills and methods of Teacher A).

In fact, the above-cited article is so fraught with error and leaps of logic (and bad faith) as to be utterly and irredeemably worthless, not too unlike the test scores upon which its false (and probably libelous) conclusions are based. But then, it began with a patently incorrect assumption. And likely had its conclusions well in mind to begin with.

May we expect in the near future an article by the same reporters on which LA TIMES journalists are "most effective" and "least effective" based on how sales of the paper are impacted by their articles and reportage?

May we expect that the reporters will be taking, say, the tests given to high school kids in LAUSD (I assume all these journalists graduated from college) or perhaps the SAT or ACT (or, Darwin forbid! the GRE) and publishing the results in the paper? How about the politicians who pushed and voted for using these tests as fair measures of a host of things they were never designed to assess? (And I include in that list not only state and local officials, but every US senator, congressperson, US Department of Education secretary, and US president from William Jefferson Clinton to George W. Bush to Barack Hussein Obama who has supported these tests as the measure of all things.)

Let's shed more sunshine on the test-based competence of our reporters and politicians. Publish and publicize their scores. Threaten, meaningfully, to hoist these folks by their own petard and we'll see some critical examination of the assumption that the tests are valid and reliable, as well as adequate measures alone of effectiveness or lack thereof. My suggestion is no worse than what politicians and reporters are doing now with kids, administrators, schools, districts, and, of course, everyone's favorite scapegoat, public school teachers.

Until such time, shame on Messrs. Felch, Song, Smith; shame on the LA TIMES. Shame on anyone and everyone who buys into the ridiculous test-mad nonsense that has this country by the throat.

p.s.: Lest someone suggest otherwise, my last official GRE scores, taken in October 1991, are Verbal 800; Mathematics 780; and Logical Reasoning 720. I feel safe in putting those scores up against those on any comparable test of the three reporters, any member of the LA TIMES staff, any current legislator in California or the United States Congress. I've spent over 30 years preparing students for various standardized tests and debunking many of the myths surrounding them.

I'll happily meet anyone on the standardized test battle ground, No. 2 pencils aready at dawn or high noon.


By Jason Felch, Jason Song and Doug Smith, in The Los Angeles Times

The fifth-graders at Broadous Elementary School come from the same world -- the poorest corner of the San Fernando Valley, a Pacoima neighborhood framed by two freeways where some have lost friends to the stray bullets of rival gangs.

Many are the sons and daughters of Latino immigrants who never finished high school, hard-working parents who keep a respectful distance and trust educators to do what's best.

The students study the same lessons. They are often on the same chapter of the same book.

Yet year after year, one fifth-grade class learns far more than the other down the hall. The difference has almost nothing to do with the size of the class, the students or their parents.

It's their teachers.

With Miguel Aguilar, students consistently have made striking gains on state standardized tests, many of them vaulting from the bottom third of students in Los Angeles schools to well above average, according to a Times analysis. John Smith's pupils next door have started out slightly ahead of Aguilar's but by the end of the year have been far behind.

In Los Angeles and across the country, education officials have long known of the often huge disparities among teachers. They've seen the indelible effects, for good and ill, on children. But rather than analyze and address these disparities, they have opted mostly to ignore them.

Most districts act as though one teacher is about as good as another. As a result, the most effective teachers often go unrecognized, the keys to their success rarely studied. Ineffective teachers often face no consequences and get no extra help.

Which teacher a child gets is usually an accident of fate, in which the progress of some students is hindered while others just steps away thrive.

Though the government spends billions of dollars every year on education, relatively little of the money has gone to figuring out which teachers are effective and why.

Seeking to shed light on the problem, The Times obtained seven years of math and English test scores from the Los Angeles Unified School District and used the information to estimate the effectiveness of L.A. teachers -- something the district could do but has not.

The Times used a statistical approach known as value-added analysis, which rates teachers based on their students' progress on standardized tests from year to year. Each student's performance is compared with his or her own in past years, which largely controls for outside influences often blamed for academic failure: poverty, prior learning and other factors.

Though controversial among teachers and others, the method has been increasingly embraced by education leaders and policymakers across the country, including the Obama administration.

In coming months, The Times will publish a series of articles and a database analyzing individual teachers' effectiveness in the nation's second-largest school district -- the first time, experts say, such information has been made public anywhere in the country.

This article examines the performance of more than 6,000 third- through fifth-grade teachers for whom reliable data were available.

Among the findings:

  • Highly effective teachers routinely propel students from below grade level to advanced in a single year. There is a substantial gap at year's end between students whose teachers were in the top 10% in effectiveness and the bottom 10%. The fortunate students ranked 17 percentile points higher in English and 25 points higher in math.

  • Some students landed in the classrooms of the poorest-performing instructors year after year -- a potentially devastating setback that the district could have avoided. Over the period analyzed, more than 8,000 students got such a math or English teacher at least twice in a row.

  • Contrary to popular belief, the best teachers were not concentrated in schools in the most affluent neighborhoods, nor were the weakest instructors bunched in poor areas. Rather, these teachers were scattered throughout the district. The quality of instruction typically varied far more within a school than between schools.

  • Although many parents fixate on picking the right school for their child, it matters far more which teacher the child gets. Teachers had three times as much influence on students' academic development as the school they attend. Yet parents have no access to objective information about individual instructors, and they often have little say in which teacher their child gets.

  • Many of the factors commonly assumed to be important to teachers' effectiveness were not. Although teachers are paid more for experience, education and training, none of this had much bearing on whether they improved their students' performance.

  • Other studies of the district have found that students' race, wealth, English proficiency or previous achievement level played little role in whether their teacher was effective.

    "In the past, too often we've just gone with gut instinct and haven't been careful about whether those things are important," said Richard Buddin, a senior economist and education researcher at Rand Corp., who conducted the statistical analysis as an independent consultant for The Times.

    Many teachers and union leaders are skeptical of the value-added approach, saying standardized tests are flawed and do not capture the more intangible benefits of good instruction. Some also fear teachers will be fired based on the arcane calculations of statisticians who have never worked in a classroom.

    The respected National Academy of Sciences weighed in last October, saying the approach was promising but should not be used in "high stakes" decisions --firing teachers, for instance -- without more study.

    No one suggests using value-added analysis as the sole measure of a teacher. Many experts recommend that it count for half or less of a teacher's overall evaluation.

    And in Los Angeles, the method can be used for only a portion of the district's roughly 14,000 elementary school instructors: California students don't take the test until second grade and teachers must have had enough students for the results to be reliable.

    Nevertheless, value-added analysis offers the closest thing available to an objective assessment of teachers. And it might help in resolving the greater mystery of what makes for effective teaching, and whether such skills can be taught.

    On visits to the classrooms of more than 50 elementary school teachers in Los Angeles, Times reporters found that the most effective instructors differed widely in style and personality. Perhaps not surprisingly, they shared a tendency to be strict, maintain high standards and encourage critical thinking.

    But the surest sign of a teacher's effectiveness was the engagement of his or her students -- something that often was obvious from the expressions on their faces.

    Study in contrasts

    On a spring day at Broadous, all eyes in Room 26 were on the white board.

    Miguel Aguilar had brought his fifth-graders to the edge of their seats -- with a math problem.

    Aguilar, a stocky 33-year-old who grew up in the area, is no showman. Soft-spoken and often stern, he doles out praise sparingly. It only seems to make his students try harder.

    "Once in a while we joke around, but they know what my expectations are," he said. "When we open a book, we're focused."

    It seems to work: On average, his students started the year in the 34th percentile in math compared with all other district fifth-graders. They finished in the 61st. Those gains, along with strong results in English, made him one of the most effective elementary school teachers in the district.

    On this day, Aguilar had invited a student to the board to divide two fractions -- a topic on the upcoming state exam. As his classmates compared notes in whispers, the boy wrote out his answer. Aguilar turned to the class.

    "Do you agree?" he asked, without hinting at the correct response.

    "Yes!" they called back in unison.

    "Good," he said softly, allowing a faint smile. "You know this."

    John Smith's students in Room 25 were studying fractions too.

    Speaking in a slow cadence, he led his class in reciting a problem aloud twice. He then called on a student slouched in the back. The boy got the answer wrong.

    "Not so much," Smith said dryly, moving on to another pupil without explanation.

    It was only 11 a.m., and already it had been a tough day: Three of Smith's students were sitting in the principal's office because of disruptive behavior. All were later transferred permanently to other classrooms.

    In an interview days later, Smith acknowledged that he had struggled at times to control his class.

    "Not every teacher works with every kid," said Smith, 63, who started teaching in 1996. "Sometimes there are personality conflicts."

    On average, Smith's students slide under his instruction, losing 14 percentile points in math during the school year relative to their peers districtwide, The Times found. Overall, he ranked among the least effective of the district's elementary school teachers.

    Told of The Times' findings, Smith expressed mild surprise.

    "Obviously what I need to do is to look at what I'm doing and take some steps to make sure something changes," he said.

    Jobs with security

    Public school students are graded and tested all the time. Schools are scored too --California rates them in an annual index.

    Not so with teachers.

    Nationally, the vast majority who seek tenure get it after a few years on the job, practically ensuring a position for life. After that, pay and job protections depend mostly on seniority, not performance.

    Teachers have long been evaluated based on brief, pre-announced visits by principals who offer a confidential and subjective assessment of their skills. How much students are learning is rarely taken into account, and more than 90% of educators receive a passing grade, according to a survey of 12 districts in four states by the New Teacher Project, a New York-based nonprofit.

    Almost all sides in the debate over public education agree that the evaluation system is broken. The dispute centers on how to fix it.

    Value-added analysis offers a rigorous approach. In essence, a student's past performance on tests is used to project his or her future results. The difference between the prediction and the student's actual performance after a year is the "value" that the teacher added or subtracted.

    For example, if a third-grade student ranked in the 60th percentile among all district third-graders, he would be expected to rank similarly in fourth grade. If he fell to the 40th percentile, it would suggest that his teacher had not been very effective, at least for him. If he sprang into the 80th percentile, his teacher would appear to have been highly effective.

    Any single student's performance in a given year could be due to other factors -- a child's attention could suffer during a divorce, for example. But when the performance of dozens of a teacher's students is averaged -- often over several years -- the value-added score becomes more reliable, statisticians say.

    The approach, pioneered by economists in the 1970s, has only recently gained traction in education.

    A small number of states and districts already use value-added scores to determine which teachers should be rewarded and which need help. This summer, one district took a harder line: Washington, D.C., schools Chancellor Michelle Rhee fired 26 teachers based in significant part on their poor value-added scores.

    Prompted by federal education grants, California and several other states are now proposing to make value-added a significant component of teacher evaluations. If the money comes through, Los Angeles schools will have to rely on the data for at least 30% of a teacher's evaluation by 2013.

    The Times found that the district could have acted far earlier. In the last decade, district researchers have sporadically used value-added analysis to evaluate charter schools and study after-school programs. Administrators balked at using the data to study individual teachers, however, despite encouragement from the district's own experts.

    In a 2006 report, for instance, L.A. Unified researchers concluded that the approach was "feasible and valid" and held "great promise" for improving instruction. But district officials did not take action, fearful of picking a fight with the teachers union in the midst of contract negotiations, according to former district officials.

    In an interview last week, A.J. Duffy, president of United Teachers Los Angeles, was adamant that value-added should not be used to evaluate teachers, citing concerns about its reliance on test scores and its tendency to encourage "teaching to the test." But Duffy said the data could provide useful feedback.

    "I'm not opposed to standardized tests as one means to helping teachers look at what's happening in their classrooms," he said.

    Supt. Ramon C. Cortines, who was appointed in late 2008 and plans to retire in the spring, acknowledged that the district could have done more with value-added analysis but was focused on other ways of improving instruction, as well as with staying solvent.

    "We have better data than anyone else in the nation -- we just don't use it well," he said. "I think it's the next step. It has to be done."

    A task force created by the Los Angeles school board to promote teacher effectiveness raised the issue in April, urging the use of value-added scores as one measure of performance.

    The task force chairman, Ted Mitchell, said the changes were long overdue.

    "I think it's simply a failure of will," said Mitchell, who also heads the State Board of Education.

    'Don't be a robot!'

    Even at Third Street Elementary in Hancock Park, one of the most well-regarded schools in the district, Karen Caruso stands out for her dedication and professional accomplishments.

    A teacher since 1984, she was one of the first in the district to be certified by the National Board for Professional Teaching Standards. In her spare time, she attends professional development workshops and teaches future teachers at UCLA.

    She leads her school's teacher reading circle. In her purse last spring, she carried a book called "Strategies for Effective Teaching."

    Third Street Principal Suzie Oh described Caruso as one of her most effective teachers.

    But seven years of student test scores suggest otherwise.

    In the Times analysis, Caruso, who teaches third grade, ranked among the bottom 10% of elementary school teachers in boosting students' test scores. On average, her students started the year at a high level -- above the 80th percentile -- but by the end had sunk 11 percentile points in math and 5 points in English.

    Caruso said she was surprised and disappointed by her results, adding that her students did well on periodic assessments and that parents seemed well-satisfied.

    "Ms. Caruso was an amazing teacher," said Rita Gasparetti, whose daughter was in Caruso's class a few years ago. "She really worked with Clara, socially and academically."

    Still, Caruso said the numbers were important and, like several other teachers interviewed, wondered why she hadn't been shown such data before by anyone in the district.

    "For better or worse," she said, "testing and teacher effectiveness are going to be linked. If my student test scores show I'm an ineffective teacher, I'd like to know what contributes to it. What do I need to do to bring my average up?"

    During recent classes observed by a reporter, Caruso set clear expectations for her students but seemed reluctant to challenge them. In reviewing new vocabulary, for instance, Caruso asked her third-graders to find the sentence where the word "route" appeared in a story.

    "Copy it just like it's written," she instructed the class, most of whom started the year advanced for their grade.

    "Some teachers have kids use new words in their own sentences," Caruso explained. "I think that's too difficult."

    She dismissed the weekly vocabulary quizzes that other teachers give as "old school."

    Down the hall from Caruso, fourth-grade teacher Nancy Polacheck was grilling her students on vocabulary, urging them to think hard about what the words meant.

    "Don't be a robot!" she said.

    Polacheck is another teacher whom Oh identified as one of her top performers. And the Times analysis suggests that the principal is right: Polacheck's students gained 5 percentile points in math after a year in her class, and 4 points in English. That put her in the top 5% of elementary school teachers.

    An animated woman with a blond ponytail flowing from the top of her head into her bespectacled eyes, Polacheck has been teaching for 38 years. The desks in her classroom are often set up like seats around a stage, with Polacheck, a self-described "drama queen," in the center.

    Her teaching style is a rat-a-tat-tat of questions, the most common of which is "why?"

    Polacheck said her colleagues at Third Street think her expectations are too high. She was reluctant to be singled out in any way, repeatedly asking a reporter why she was being interviewed.

    "In the past, if I were recognized, I would become an outcast," said Polacheck, who eats her lunch alone in her classroom. "They'd say, 'She's trying to show off.'"

    A long process

    As the district was appointing the task force and seeking federal dollars, some enterprising principals in L.A. schools began making back-of-the-envelope assessments of teachers using raw test scores.

    One clear lesson so far: Finding the least effective teachers is only the first step in a long process.

    In 2008, officials at Sunrise Elementary in Boyle Heights, one of 15 campuses under Mayor Antonio Villaraigosa's control, identified several teachers whose student scores were sliding.

    The mayor's Partnership for Los Angeles Schools gave them math and literacy coaches, more feedback and the opportunity to watch their more effective peers, said Angela Bass, the group's former superintendent of instruction.

    Of three Sunrise teachers who ranked in the bottom 10% districtwide, just one has dramatically improved, according to the Times analysis.

    Bass acknowledged that it could take years for foundering instructors to improve, if they do at all. In the meantime, about 20 students a year will continue to sit through their classes.

    "It's tragic," Bass said. "It means we've failed them."

    Miko Dixon, the principal at Topeka Drive Elementary in Northridge, took a tougher approach. Upon starting the job in 2009, she said, she identified four highly ineffective teachers. One had decided, with the approval of the previous principal, to keep his second-graders as they moved to third grade, ensuring two years of poor teaching in a row.

    "It's criminal," Dixon said. "If you get a bad teacher in second and third grade, you're doomed."

    Dixon has begun trying to remove the four teachers, a painfully slow process in California. It's far more likely that they will feel the pressure and transfer to another school, she said.

    "That's not right," Dixon said, "but it's reality."

    Parental trust

    For now, parents remain mostly in the dark.

    Even the most involved mothers and fathers have little means of judging instructors other than through classroom visits and parking lot chatter. Others don't even have time for that.

    Without reliable information, it comes down to trust. Which instructor a child gets is usually decided behind closed doors by principals and teachers, whose criteria vary widely.

    "Mi nino, all his teachers are good," said Maura Merino, whose son Valentin Cruz was in the fifth-grade class of John Smith, the low-performing Broadous teacher, last school year. "He never had a problem. Everything is OK."

    Merino said it's hard for her to tell the difference between teachers because she doesn't speak English. If she knew her son was assigned to a struggling teacher, "I wouldn't know what to do," she said, speaking in Spanish. "But I would try to get him to the best."

    In a conversation after school one day, several Broadous teachers, including Aguilar and Smith, said parents should have the chance to see how teachers measure up.

    They "might be more empowered to demand a good teacher," said teacher Eidy Hemmati. And it might keep teachers "on their toes a little bit more," Smith said.

    But many others say it would be impossible to accommodate every parent's desire for the best teacher, and publicizing disparities would only turn one educator against the other.

    Broadous Principal Stannis Steinbeck refused even to discuss the differences among her instructors, hinting at the tensions that might arise on staff.

    "Our teachers think they're all effective," she said.




    Times data analyst Sandra Poindexter contributed to this report.

    — Jason Felch, Jason Song and Doug Smith
    Los Angeles Times





    This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.