Orwell Award Announcement SusanOhanian.Org Home

NCLB Outrages

The Education Establishment Keeps Circling the Wagons

But what if "scientifically-based reading" is just a crock of crap? Maybe the teacher and the children would be better served by asking this question--instead of following the advice below--and memorizing the definition--as promulgated by No Child Left Behind.

Don't get buried under a mountain of research

Use these strategies to sort through the pile and find the information you need

Consider this scenario: It is school improvement planning time--again. While preparing a professional development plan based on literacy, you begin an intense search for articles and information that you can use to align this plan with both your state and national goals. A priority in this search is material that will meet the Reading First stipulations of the No Child Left Behind Act of 2001, the $900 million program aimed at ensuring that the nation's children are proficient readers by the end of 3rd grade. As you begin digging into the pile of articles you have accumulated, the sheer volume of information seems overwhelming. Where do you start?

Becoming a sophisticated consumer of research on literacy is a continuing process, not a one-time accomplishment. It begins--as does any journey into new territory--with adjustments and attempts to comprehend new information that may seem a foreign language. Then it requires verbalizing what you don't know, asking questions. Finally, it requires a keen sense of inquiry, sniffing out the best sources of information to address those questions.

Here are some practical tips for your journey of applying the right type of research to translating research into classroom practice.

Tip 1:
Savvy readers comprehend research language.

First, learn the core definition of scientifically based research, which appears in the No Child Left Behind legislation, under Reading First (Title I, Part B, Subpart 1, Section 1208):

The term "scientifically based reading research" means research that--
(A) Applies vigorous, systematic, and objective procedures to obtain valid knowledge relevant to reading development, reading instruction, and reading difficulties; and
(B) Includes research that--
i. Employs systematic, empirical methods that draw on observation or experiment;
ii. Involves rigorous data analyses that are adequate to test the stated hypotheses and justify the general conditions drawn;
iii. Relies on measurements or observational methods that provide reliable and valid data across evaluators and observers and across multiple measurements and observations; and
iv. Has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review.

Within and related to this definition are a number of terms that warrant further translation:

Observational methods: Methods researchers use to document events, behaviors, and surroundings.

Data analyses: Using the artifacts of instruction to help see more clearly where students are, where they are going, and what patterns are emerging.

Hypothesis: A statement of prediction of the results of a research project.

Professional wisdom: "The judgment that individuals acquire through experience. ... Increased professional wisdom is reflected in numerous ways, including the effective identification and incorporation of local circumstances into instruction" (Whitehurst, 2002).

Empirical evidence: "Scientifically based research from fields such as psychology, sociology, economics, and neuroscience, and especially from research in educational settings. ... Objective measure of performance used to compare, evaluate, and monitor progress" (Whitehurst, 2002).

Evidence-based education: "The integration of professional wisdom with the best available empirical evidence in making decisions about how to deliver instruction" (Whitehurst, 2002).

Tip 2:
Sophisticated readers of research ask the right questions.

A dozen years ago, S. Jay Samuels and Alan E. Farstrup offered an insight about research that remains valid today. They wrote, "Research is not a collection of ready-made answers to instruction-related questions, waiting to be claimed by eager and trusting teachers. It is a resource that can provide direction and substance for making instructional decisions when it is approached with purpose and caution" (Farstrup & Samuels, 1992, p. 1).

As consumers of research, educators must critically inspect materials that claim to be research-based. Educators must bring to the table qualities that enhance this process-- qualities such as a well-developed sense of inquiry, experience in the field, and commitment to authentic learning. Evaluating research as a smart consumer means knowing how to interpret the claims made related to being "scientifically based." When reviewing materials that purport to provide research-based evidence, take the following steps:

First, determine what kind of document it is. Generally, there are three categories of documents:

How-to documents, which describe, step-by-step, the process of implementing a strategy or program.
Descriptive documents, which generally describe the unqualified success of one program or another. Such documents may include a scenario, individual program, initiative, curriculum, strategy, plan, or method.
Research documents, which generally include some or all of the following: literature review, results of a study or survey, thesis supported by evidence. Such documents likely will include graphs, tables, charts, statistics, or other data to support the thesis.

As informative as how-to, descriptive, and some research documents are, a closer look is needed to interpret whether the evidence provided is sufficient and valid , and whether the evidence is transferable to your classroom or school. Sound research should be able to be used in the classroom to assist with curriculum, instruction, and assessment.

Feuer, Towne, and Shavelson (2002) state, "No method is good, bad, scientific, or unscientific in itself: Rather, it is the appropriate application of method to a particular problem that enables judgments about scientific quality" (p. 8).

Second, ask these questions:

WHO conducted this research? Does the organization or individual possess credible background to address the research topic with an authoritative voice? Is there reason to believe that the organization would have any bias regarding the topic that might make the research results questionable?

WHAT is being researched? Is the research topic focused sufficiently so that the data gathered can support feasible results? Is the information interpreted without bias?

WHEN was the research conducted? Classic research studies can have as much validity as recent research studies. However, the timeliness of the results is important, and it should influence how the reader applies the results of the research. If the research is brand-new, does it pay attention to the studies and findings that came before?
WHERE was research done? Is the number of individuals involved in the study enough for a credible measure, given the topic studied? Does the research cover people and places like your classroom or school?

WHY was the research conducted? Remember: context, context, context. If the research was conducted to prove the effectiveness of a program, or sponsored by persons who could benefit from positive results, this reason needs to be taken into consideration. Is there any other research that is relevant to this piece? If so, are the researchers speaking with the same voice? If not, why do the pieces disagree? Research is a community enterprise. Communal work and expertise is needed to validate findings. Research that provides answers to questions should not be carried out in isolation, but in a community of inquiry.

HOW was the data gathered and analyzed? Is the methodology sound? What process was used to develop appropriate questions and to determine how to organize the data and how to analyze the data? Is the solution connected to the research? Is it clear that the instrument or intervention made the changes? Again, there are resources that can help you with this.

HMMMM? Trust your own instincts. Remember, you bring useful perspective and knowledge to the table. If the research seems tainted or suspect, that is reason to pause and ask, "Why?"

Tip 3:
Inquisitive readers find reliable information.

The search for reliable sources begins with taking a close look at the existing research base. Credible sources are very important. Many of the same questions suggested to examine research also can be asked of information sources.

If you are beginning to investigate a specific content area, such as reading, here are some supplemental questions: What information can be found? What information is missing? What information is debated within the community? If you ask yourself these questions, you will begin to feel comfortable that you have identified the most current and relevant research.

Starting Points:

The newly established What Works Clearinghouse (www.w-w-c.org) is a web site that helps educators make choices based on scientifically based research. The site, sponsored by the U.S. Department of Education, is being updated continuously and will offer information on various topics.

Handbooks of research are available in many content areas. These publications typically are edited by the leading scholars of the discipline, and they feature powerful and relevant research.

Scholarly journals typically have a review process for material published. The complexity and thoroughness of this process varies from journal to journal. The description of these processes can be found in the journals themselves. The articles published in journals typically offer descriptions of new research and the studies' results.

A list of "Places to Begin Building Knowledge" organized by topic in the box below is a suggested starting point.

Tip 4:
Reading professionals take action.

Scientifically based reading research provides a framework through which we can continue to engage in investigation--and in instruction based on evidence. Begin to learn the language, ask the questions, and seek reliable sources for answers. Collect data to support or refute assumptions. As we bring more research into our schools and classrooms, this framework assists us in implementing new practices and helps us become more sophisticated consumers of literacy materials.

Educators, policy makers, and professional development providers are in the position--now more than ever--to contribute to the knowledge base and further change the landscape of American education. Although there are no quick steps for translating research into practice, it's time to begin the journey.


Farstrup, A. & Samuels, S.J.(Eds.) . (1992). What research has to say about reading instruction. Newark, DE: International Reading Association.

Feuer, M.J., Towne, L., & Shavelson, R.J. (2002, November). Scientific culture and educational research. Educational Researcher, (8)31, 4-14.

No Child Left Behind Act of 2001, Pub. No. 107-110, 115 Stat. 1425. (2002). Retrieved December 20, 2002, from www.ed.gov/legislation/ESEA02/.

Whitehurst, G.J. (2002). Evidence-based education [Slide presentation]. Retrieved December 20, 2002, from www.ed.gov/offices/OESE/SASA/eb/evidencebased.pdf.

Collection of reading materials

State-Specific Reading First Committees. Under the new legislation, states submit applications to the federal government for funding. In these applications, the states must lay out a comprehensive plan that details program information, professional development information, and assessment information. Because each state's plan differs, educators must pay close attention to their state's specific plan. Check the state's web site to see what resources are available.

NRP Report. The National Reading Panel report, Teaching Children to Read, is a meta-analysis of reading research. It offers specific information that fits the scientifically based reading research criteria. Both an executive summary and the full report are available online at www.nationalreadingpanel.org/Publications/summary.htm.

CIERA and NIFL Report. Put Reading First is a publication co-sponsored by the Center for Improvement of Early Childhood Reading Achievement (CIERA) and the National Institute for Literacy. It details specific instructional implications. Strategies are suggested for teaching the five essential elements of reading. This publication is available online at www.nifl.gov/partnershipforreading/publications/Cierra.pdf.

Other helpful books. The International Reading Association (IRA) has published many helpful books. Two suggestions are:

What Research Has to Say About Reading Instruction (3rd edition), edited by Alan E. Farstrup and S. Jay Samuels, is a collection of articles on research and reading instruction.
Evidence-Based Reading Instruction--Putting the National Reading Panel Report to Work is a collection of articles published in various IRA journals.

Places to begin building knowledge

Here are several credible sources that can be used as starting points on your quest:


Fetterman, D.M., Kaftarian, S.J., & Wandersman, A. (Eds.). (1996). Empowerment evaluation: Knowledge and tools for self-assessment and accountability. Thousand Oaks, CA: Sage Publications.
Fetterman, D.M. & Pitman, M.A. (Eds.). (1986). Educational evaluation: Ethnography in theory, practice, and politics. Thousand Oaks, CA: Sage Publications.
Maruyama, G. & Deno, S. (Eds.). (1992). Research in educational settings (Applied Social Research Methods Series, Vol. 29). Thousand Oaks, CA: Sage Publications.
McNamara, C. (1998). Basic guide to program evaluation. Retrieved December 20, 2002, from www.mapnp.org/library/evaluatn/fnl_eval.htm.
Patton, M.Q. (1990). Qualitative evaluation and research methods (2nd ed.). Thousand Oaks, CA: Sage Publications.
W.K. Kellogg Foundation. (1998, January). W.K. Kellogg Foundation evaluation handbook. Battle Creek, MI: Author.


Booth, W.C., Colomb, G.G., & Williams, J.M. (1995). The craft of research (2nd ed.). Chicago: University of Chicago Press.
Committee on Scientific Principles for Education Research. (2002). Scientific research in education (R.J. Shavelson & L. Towne, Eds.). Washington, DC: National Academy Press.
Gall, J.P., Gall, M.D., & Borg, W.R. (1999). Applying educational research: A practical guide (4th ed.). New York: Allyn & Bacon.
Merriam, S.B. (1997). Qualitative research and case study applications in education. San Francisco, CA: Jossey-Bass.
Mertens, D.M. (1997). Research methods in education and psychology: Integrating diversity with quantitative and qualitative approaches. Thousand Oaks, CA: Sage Publications.
Stake, R.E. (1995). The art of case study research. Thousand Oaks, CA: Sage Publications.
Vogt, W.P. (1999). Dictionary of statistics and methodology: A nontechnical guide for the social sciences (2nd ed.). Thousand Oaks, CA: Sage Publications.


Barr, R., Kamil, M.L., Mosenthal, P., & Pearson, P.D. (Eds.). (1996.) Handbook of reading research (Vol. 2). Mahwah, NJ: Lawrence Erlbaum Associates.
Farstrup, A. & Samuels, S.J. (Eds.). (2002). What research has to say about reading instruction. Newark, DE: International Reading Association.
Kamil, M., Mosenthal, P., Pearson, P.D., & Barr, R. (Eds.). (2000). Handbook of reading research (Vol. 3). Mahwah, NJ: Lawrence Erlbaum Associates.
Learning First Alliance. (2000). Every child reading: A professional development guide. Baltimore, MD: ASCD.
National Reading Panel. (2000). Teaching children to read: Reports of the subgroups. Washington, DC: Author. Retrieved December 20, 2002, from www.nationalreadingpanel.org/Publications/subgroups.htm.
Pearson, P.D., Barr, R., Kamil, M.L., & Mosenthal, P. (1984). Handbook of reading research (Vol. I). New York: Longman.
Snow, C.E., Burns, M.S., & Griffin, P. (Eds.). (1998). Preventing reading difficulties in young children. Washington, DC: National Academy Press.

— Danielle Carnahan & Michele Fitzpatrick
Don't get buried under a mountain of research
Journal of Staff Development
Spring 2003


This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.