Orwell Award Announcement SusanOhanian.Org Home


Goldwater Institute Report on Special Education

Susan Notes: Although this section is titled Research that Counts, here's a critique on research that jiggles the numbers. Education needs more of such deconstructions of pseudo-science.


by ASBA Research Analyst Michael Martin

The Goldwater Institute policy report Race and Disability: Racial Bias in Arizona Special Education is yet another polemic that serves merely to give a pseudo-logical patina to their voucher argument. They heavily footnote their report but almost all the footnotes are to their compatriot polemicist Jay Greene.

Fundamentally their argument rests on the belief that minorities are referred prejudicially to Special Education in order for school districts to receive extra funding. Their report ignores that in January, 2002, the National Academy of Sciences' Commission on Behavioral and Social Sciences and Education released its report titled "Minority Students in Special and Gifted Education" that discounted the allegation that educators were prejudicially referring minority children to special education. On page four of the NAS Executive Summary the committee noted: "For example, research that has compared groups of students who are referred by teachers find that minority students actually have greater academic and behavior problems than their majority counterparts."

There does not appear to be a single example of research in the Goldwater Institute report from a legitimate "juried" publication where articles have to be screened by people with expertise. The numbers they generate in this report are irrelevant and they seem unaware that "statistically significant" does not mean logically significant. The report spends considerable effort speculating about issues that explain less than 12% of the variation in their data.

But even more significant is that their conclusions are not supported by the data of their study anyway. They state "Even after controlling for school spending, student poverty, community poverty, and other factors, this study finds a common pattern of predominantly White public school districts in Arizona placing minority students into special education at significantly higher rates."

I did not find anywhere in the report where they controlled for student poverty, or community poverty, let alone "other factors." Their "percentage economically disadvantaged" was presumably the "free and reduced lunch" value which is a crude measure of poverty but it tends to be correlated with their other two independent variables and therefore is not statistically "independent" in the sense that it provides any controlling value.

More importantly, their three independent variables hardly explained anything at all. The R-squared value they refer to is a measure of the percent of any variation that is predicted by knowing the independent variables. The 12% R-squared they cite in their report means that 88% of the variation in the rate that minorities are referred to Special Education is explained by other unaccounted for variables not in their report(in the White case the other variables explained 97% of the differences). Even the 12% likely results from some collinearity with the study variables.

In other words, when you are trying to predict one thing by knowing another thing, it may be just coincidence or it may be because the thing you know tends to have some relationship with the real explanation that you haven't accounted for. An example is that belt size can have some accuracy in predicting height because belt size is collinear with weight and tall people tend to be heavier than short people. When the R-squared is in the range of 12% it indicates the study variables probably have little to do with the dependent variable.

In social science research it is difficult to find any two things that do not show some spurious correlation, so twelve percent for three variables has almost no significance. The authors use the words "statistical significance" but this does not mean "logically significant." Statistical significance is a technical term for whether the number of cases (N) is mathematically sufficient. You can always achieve statistical significance by increasing the number of cases. It has no meaning in terms of whether the finding logically makes any sense.

But even though their three independent variables COMBINED only showed a 12% explanatory value they never demonstrated which of those three variables provided the explanatory power. It is entirely possible that virtually all of the explanatory power was produced by their spending per pupil value alone. They never checked for this. The usual method of doing multiple regression is to find variables that result in a significant explanatory value (lots more than 12%), and then enter the most powerful one first and then enter the next powerful one to see if the explanatory power increases substantially over the first variable by itself (i.e. stepwise regression).

I say substantially because it is well known mathematically that each new variable (even one totally unconnected to the issue) will increase the explanatory power (i.e. R-squared) simply because some of its variation will randomly occur in concert with the dependent variable. Thus by sequentially entering the independent variables you can see how much difference they each make in adding to the explanatory value.

The regression they show here is so low they cannot realistically state in their conclusion that it has any explanatory power in regard to their claim that districts with higher percentages of Whites have higher percentages of minorities in Special Education, but more importantly, they cannot state which of their variables has what little power they show. But that doesn't stop them.

In one paragraph of their conclusion they state "Mislabeling students costs Arizona an estimated $50 million each year." Where in their report did this "estimate" come from? It actually didn't come from their report, it came from another poorly designed and mathematically challenged "report" with (page 11) fellow polemicist Jay Greene and Forster "theorizing" that "bounty" systems increase funding.

The Greene and Forster "report" was little more than speculation about an unexplained marginal difference in spending among states that use different funding models and had absolutely nothing to do with "mislabeling." The Greene and Forster "report" cannot conclude that "mislabeling students" was the source of the spending difference and therefore it provides zero justification for this report making that a conclusion.

It is not even clear that the marginal difference they cite has any real significance. An Education Week article addressed their claim and it stated:


"Special education enrollment nationwide grew from 10.6 percent of all students to 12.3 percent between 1991- 92 and 2000-01. During that time, special education enrollment grew from 10.6 percent to 12.6 percent in the 33 states (and the District of Columbia) that had bounty systems at that time. The enrollment level in the states with lump- sum systems grew from 10.5 percent to 11.5 percent."

Notice first that we are talking about minor differences here: the "bounty" states grew two percent points while the lump sum states grew one percentage point. There is no data to qualify this: were students under-served in states that did not have an "incentive?" Notice second that the growth in the "bounty" states is almost the same as overall growth: 12.6 versus 12.3 percent. This indicates that most of the Special Education students are in the bounty states. The non-bounty states therefore may not have sufficient Special Education students to make additional funding (i.e. "bounty") significant.

In addition, states shifted from non-bounty to bounty programs during the period referenced and it is possible they had these marginal differences before the lump sum states switched but the report does not consider that. However, the reality is that the bulk of the money goes to the 11.5% in both groups of states, the "bounty" states pay only marginally more (1.1%) and we have no data on the demographic characteristics of the states in either case. What we actually have is a minor variation that allowed flights of fancy by researchers unencumbered by logic or data.

The Goldwater Institute report cites a minor difference in the averages of two groups of states that were not matched demographically or otherwise compared so that the differences represent any meaning. They then speculated about this unexplained difference and claimed ex nihilo that it occurred because of a funding method, but the funding amount difference was from 1991-92 to 2000-2001 while the change in funding method in the states occurred at various times subsequent to 1997 (page 11) and therefore could have occurred before the switch. An ex nihilo concoction regarding an unexplained difference is then used to derive a non sequiteur estimate. There was absolutely no data in this report about mislabeling of students and the National Academy of Sciences indicated it is a myth.

Then in the next paragraph of their conclusion the Goldwater Institute lauds the corruption riddled McKay voucher system in Florida, but they had no data about it in the report, and they state "it has proven to be at least revenue-neutral and could save Arizona money." To my knowledge it hasn't proven anything but you could speculate that it "could" do anything if you don't require any logic or supporting data. This statement does not belong in their report, let alone in their conclusion.

The purported logic behind their report is that low-income students typically have higher Special Education needs than more affluent students but that minority students in mostly minority districts show fewer referrals than minority students in districts with higher proportions of Whites, therefore indicating discrimination. The only data in their report supporting their conclusion shows that a greater proportion of Hispanic students in Arizona receive Special Education services in districts where they are a smaller minority. Their data shows that this is not true for African-Americans nor White students although they claim it is true for African-Americans.

Looking at their Figure 1: Disability Rates by Ethnic Group in Arizona Schools, their interpretation is clearly faulty. You actually have three primary trend lines. First, the white trend line tends to follow the expected model: higher Special Education referrals in low percentage White (and presumably low income but it is important to note that these charts do not show affluence) districts but declining in more presumably affluent districts.

Clearly the trend-line for African-American students has three of the four data points following the same trend as the White data. Only the first data point for African-Americans is clearly an anomaly that needs to be explained. However, this was a data point for African-American students in schools where less than 26% of the students are White. Those non-White districts likely are numerous Indian reservation and Mexican-border schools where less than 26% of the students are White, but those would have very few African-American students. There are not very many African-American students in Arizona proportionally and most live in inner-city urban areas. Thus even fewer live in districts where Whites are also a minority. These may be very small anomalous districts and the anomalous data-point can be dismissed as irrelevant.

The Hispanic line is clearly the only one that runs the way the Goldwater Institute alleges: higher percentage White populations with higher percentage Hispanic Special Education referrals. But there are three logical explanations for this pattern.

First, Hispanics in Arizona tend to have a large component of illegal aliens. They can attend public schools but they are not eligible for indigent health care (AHCCCS, etc.). As a consequence, IF the greater percentage White also means greater affluence THEN it is highly likely that greater affluence means greater affordability of health care and thus diagnostic services so you would expect greater numbers of Special Education referrals. When they attend schools with fewer White students they may be small rural schools less likely to have healthcare experts available for diagnostics.

Second, schools that have small percentages of White students would tend to be Reservation districts and districts along the border with Mexico. Both of these have language and cultural differences that might reflect differences in approaches to Special Education. In addition, those districts tend to lack sophisticated services of all kinds. Those districts are distinctly different from other districts because they serve unique populations in unique circumstances. For example, in the Navajo Nation the population is widely dispersed and medical care is often provided by traditional Medicine Men. As a consequence, you would expect Special Education referrals to be lower in those districts.

Third, you have to be very careful to note that the Goldwater Institute chart refers to the percentage of districts and not the percentage of students. Approximately 70% of the students attend 10% of the districts in Arizona. As a consequence, the many rural districts with small numbers of students are a larger percentage of districts than the few urban districts such as Tucson, Mesa, Deer Valley, Paradise Valley and Peoria which have enormous numbers of kids and probably most of Arizona's Hispanics in absolute numbers. Consequently, with many small districts the percentages can be distorted very easily by small numbers. So quoting numbers about districts may have little or nothing to do with numbers about students. I'm not sure their chart has anything of relevance to the topic.

The Goldwater Institute report becomes a little far-fetched at the end. The idea that providing additional funding to cover the additional costs of Special Education is a "bounty" is merely alleged and no data is provided. In reality, the cost of the average Special Education student would pay for TWO regular education students and the federal funding is supposed to be 40% but they actually only contribute a fraction of that. As a consequence, each student put into Special Education requires reducing funding available for regular education.

In affluent districts the percentage of students in Special Education is lower and their funding is typically higher, therefore they can afford to subsidize Special Education. Conversely, in low-income districts the number of students actually needing Special Education exceeds the ability of the districts to subsidize them. The idea that there is a "bounty" available for each Special Education student is absurd. The increase in revenue does not offset the increase in costs.

The Goldwater Institute report therefore falsely interprets the data that it displays in a meaningless chart and illogically references a speculative report based on insignificant marginal data that produces little or no explanatory variables to make flights of fancy from an ex nihilo concoction regarding an unexplained difference used to derive a non sequiteur estimate that a voucher program in another state provides better results than their erroneous interpretation of Arizona's "bounty" system for Special Education. In short, it's a hoax.



Michael T. Martin is Research Analyst at
Arizona School Boards Association

— Michael T. Martin


http://www.azsba.org/hoaxgispeced.htm


INDEX OF RESEARCH THAT COUNTS


FAIR USE NOTICE
This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.