Orwell Award Announcement SusanOhanian.Org Home

Bubble Students and Teaching to the Statistic

Publication Date: 2004-09-23

A couple of years ago, Dave Posner wrote a letter to the editor that circulated on teacher lists, with people arguing whether it was satire or not. Dave suggested that teachers could improve the statistical performance of their classes by giving special attention to selected students. He noted that the particular choice of students to favor would depend on what statistical measure was being used. For example, if the mean is used then the teacher would probably get the most bang for her time by working with the strong students and ignoring the weak ones. If, on the other hand, the median were used then the smart teacher would completely ignore the weakest, give less attention to the strongest and focus on the middle. In general the teacher evaluates each student with respect to improvement of the given metric per time spent with the student. She then allocates most of her attention to students with the highest return per minute spent.

I admit that I had to write and ask--to be sure that his tongue was halfway through his cheek. After all, some version of what he was advocating was being practiced in schools across the land. As he mentioned in ensuing correspondence, "This country is far and away the world leader in innovation and technology. Things are not broken on the 'high end of the curve.' Where things actually seem to be broken is on the 'low end of the curve,' and those are the people most victimized by this process. It's completely ass backwards."

Dave also asked why the intellectual community remains so silent about this outrage. Why indeed?
(Our exchange was published as "Is Satire Possible?" Phi Delta Kappan, December 2002.)

Now Dave is at it again. See below.

Some people were appalled at the idea of a teacher favoring some students over others but optimizing the statistic is the whole point of statistical quality control -- the same process that has been used to improve the quality of automobiles and computer chips. You define a metric for whatever process you're optimizing and rigorously optimize your use of resources to maximize that metric. In the case of teaching I call this strategy "teaching to the statistic." It's a step beyond mere teaching to the test. Rather than treating the class as a collection of separate individuals, the teacher treats the entire class as a single statistic-producing system.

Given the uproar my letters caused I never imagined the ideas would be put to use, so you can imagine my delight when my wife, who is a third grade teacher, was invited, even required, to attend an in-service workshop on isolating and targeting "bubble students."

I have heard that school systems all over the country are adopting this strategy and there is lots of industry support in the form of consultants and analytical software.

So who are the bubble students? They're exactly the students who yield the highest return for minutes invested. Here's how it works. The current metrics, which I believe were formulated originally in California for the "Academic Performance Index", are based on the percentile rankings of a class. The percentile rankings are divided into some number of ranges -- in California there are five called quintiles -- and each student is assigned a score based on what range he or she falls in. The raw score for the class is then computed as a weighted sum of the student range values.

The key observation is that all students in a given range contribute the same value to the score. So for example, in California, a student with a percentile rank of 45 is equivalent to a student with a percentile rank of 59. Now, if you are the teacher, which of these students is more deserving of your attention? Well, if you work hard with the student with rank 45 maybe you can get him up to 50 or even 55. But that doesn't help! On the other hand if you can bring up the rank 59 student just two percentile points she will jump a level. The rank 59 student is a bubble student. In general the bubble students are the ones around the scoring boundaries and the school systems are training teachers to find and focus most of their attention on the bubble students.

Now I confess I don't have a clear model of how well this strategy will work, but the point is that schools have bought into the model of statistical quality control and they're trying! I am confident that in time our teachers will be able to produce some of the highest statistics in the world!

The point I've been trying to make in my little satires is that there are real engineering (never mind ethical) problems in applying a manufacturing model to human beings. In manufacturing you are attempting to apply uniform processes to uniform materials to achieve uniform results. In manufacturing you have a clear specification of desired performance and unambiguous, precise, and easily applied measures of compliance. In manufacturing you weed out and eliminate defective materials and parts as early in the process as possible. Statistical quality control depends on a process of continuous sampling of materials and products at each stage of the process.

Sampling, rather than global testing works because of the uniformity assumptions. For the same reason it is possible to identify success in manufacturing with a statistical measure and thus bend all your efforts to improving the statistic.

In our educational "factories" we do not start with uniform materials nor do we expect or is it even desirable to achieve uniform results. In education we cannot weed out and eliminate defective parts. None of the assumptions above apply. Yet we are imposing a framework intended for manufacturing on the schools and pretending it works. Being good little workers, the school administrators and their employees are doing what they're told: improve the statistics!

Surprisingly, what I can't seem to get most people to understand is that statistics are not real!

Statistics do not go out and hold jobs and have families and vote and make decisions and all the other things that real human beings do. Surely that student at rank 45 is just as much a real human being as the student at rank 59 but because he is less likely to contribute to improving the class statistics he is treated as less of one. All the players in this completely whacked out system believe they are doing their jobs. Thus the school systems are shameless about trying to manipulate the curve because they think that's what they are supposed to do and I guess it is!

I feel like Charlton Heston in Planet of the Apes: "This a mad house!"

DAVE POSNER, whose wife teaches third grade in a public school, is a former professor of mathematics and co-founder and chief technical officer of Encirq Corporation, San Francisco.

This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.