Orwell Award Announcement SusanOhanian.Org Home

Who's Who and What's What: A Scoring Guide for NAEP, the Outfit Claiming to be The Nation's Report Card

Susan Notes: This is the first of a two-part series on NAEP. This part provides the background on the Governing Board. Why should you care? There is a huge corporate-politico push to make NAEP the nation's test, pushing a national curriculum.

You have been in the village a few days and already think you know everything better than everyone here.

Franz Kafka, The Castle

Part 1 Who's On First? In the flurry of NAEP's releasing data and the media�s tossing out accusations about state standings, how many people know what NAEP is, who sits on its board, who gives them advice? How many people look at the reading passages on which the reputation of public education in the U. S. rests?

What's NAEP? The National Assessment of Educational Progress (NAEP), which bills itself as "the Nation's Report Card," defines itself as "the only nationally representative and continuing assessment of what America's students know and can do in various subject areas." They claim, "NAEP measures the reading comprehension of students in grades 4, 8, and 12." Others disagree. Noted researcher Gerald Bracey told the Baltimore Sun that "NAEP achievement levels are no damn good." More about this later.

Under the current structure, the Commissioner of Education Statistics, who heads the National Center for Education Statistics in the U.S. Department of Education, is responsible by law for carrying out the NAEP project. The National Assessment Governing Board (NAGB), appointed by the Secretary of Education but independent of the Department, sets policy for NAEP and is responsible for developing the framework and test specifications that serve as the blueprint for the development of the assessment instruments and determines the content to be assessed. Got that? The NAEP reading assessment is based on the reading framework; the framework is determined by "a framework development process;" the National Assessment Governing Board (NAGB) is in charge of the framework. Read that last sentence to the cadence of the House that Jack Built. Or maybe "Drybones."

Part 1 NAEP: the People

In creating an independent Governing Board, Congress established a system of checks and balances for NAEP--NAGB, which sets policy; NCES (National Center for Educational Statistics), which administers the program; and test contractors, who develop and carry out the assessments. To make sure NAGB is widely representative, the law requires that the Board be composed of:

  • Two Governors, or former Governors, of different political parties
  • Two state legislators, of different political parties
  • Two chief state school officers
  • One superintendent of a local education agency
  • One member of a state board of education
  • One member of a local board of education
  • Three classroom teachers representing the grade levels (4th, 8th, and 12th) at which the National Assessment is conducted
  • One representative of business or industry
  • Two curriculum specialists
  • Three testing and measurement experts
  • One nonpublic school administrator or policymaker
  • Two school principals, one elementary and one secondary
  • Four additional members who are representative of the general public, including parents
  • The Director of the Institute of Education Sciences (ex-officio)

  • And a partridge in a pear tree.

    National Assessment Governing Board (NAGB) Responsibilities
    As specified in P.L. 107-110, the Board's responsibilities include:
  • Selecting subject areas to be assessed
  • Developing appropriate student achievement
  • Developing assessment objectives and test specifications that produce an assessment that is valid and reliable, and are based on relevant widely accepted professional standards
  • Designing the methodology of the assessment
  • Developing guidelines for reporting and disseminating results
  • Developing standards and procedures for regional and national comparisons
  • Approving all cognitive and noncognitive NAEP items
  • Taking appropriate actions needed to improve the form, content, use, and reporting of results

  • In the systems of checks and balances, the President's brother was appointed to NAGB. And some of the other brief bios of NAGB members are worth considering. Information collected from the Internet is offered without editorial comment. None is needed. The reader can ponder who might offer education expertise, who's a political flak, and who might be grinding an axe for purposes other than those benefiting children. NAGB members on the Assessment Development Committee are indicated with an asterisk.

    Darvin M. Winick, Chair, President, Winick & Associates, Dickinson, Texas, holds a Ph.D. in Organizational Psychology. Winick helped organize Texas Business and Education Coalition (TBEC) and Texans for Education, and was Chairman of the Research Advisory Committee of the Texas Educational Economic Policy Center that set out the framework for the current Texas Public School Accountability System. In 1989, Dr.Winick served as volunteer Chief of Staff for the Governor�s Task Force on Education and is currently an Advisor to the Governor�s Business Council. In An Introduction to Records at the Texas State Archives, 1854-2000 (bulk 1995-2000), Winick is singled out, along with Sandy Kress as a consultant on education.


    Testifying at a Houston reading conference in 1997 as a representative of the Governor's Business Council, Winick noted: "As a professional psychologist and a long-time research manager, I was frankly surprised at the wide range of quality that we found. We're going to emphasize, as we go forward, research, evaluation on programs. I think that's a well-founded need. Our search identified a group of highly competent researchers and teachers, including those associated with the National Institute of Child Health and Human Development. We were pleasantly surprised and especially proud that the identified group included Houstonians: Barbara Foorman, David Francis, and Jack Fletcher."


    In a 1999 Progress Report the National Reading Panel details Winick's distress at multiple methods for teaching reading and his strong stand that "Only experimental evidence should be used to set a high standard for future research."


    *Sheila M. Ford, Vice Chair, Former Principal, Horace Mann Elementary School, Washington, D.C., holds an MA in supervision and administration. Valerie Straus featured her in a positive piece in the Washington Post.

    Ford is Director of the Early Childhood Demonstration Center; proposal developer for Early Childhood Education Initiative; Director of the Geography Technology Integration Project; Administrator of the Science Innovation Comprehensive Reform Initiative; and proposal developer for the Character Education Initiative. Under her leadership, Horace Mann was named a U.S. Department of Education Blue Ribbon School in 1990, 1994, and 2003; in 1995, the school was selected as one of the top 50 in the nation by Redbook Magazine. Ms. Ford has received many awards and honors, including a 1999 Fulbright Memorial Fellowship. She was appointed to the Board of Directors of the Carnegie Institution for Science in 1998.

    In an NAGB press release, 7/2//03, Ford emphasized, "For decades people have used the political cry of improving education. Now I think finally that's taking place. The No Child Left Behind Act, which the President signed 18 months ago, was a completely bipartisan bill. There were about as many Democrats supporting it as Republicans, and it took good ideas from both sides of the aisle. One of those ideas is using data to bring about change. Another is setting high standards and focusing our efforts on achieving them. Another--and I hope our leaders will stick to this--is marshalling the resources needed to get results."

    *Francie Alexander is currently Chief Academic Officer, Scholastic, Inc. Ms. Alexander is the spokesperson and advisor for Scholastic's Home Phonics Reading Program and is the author of Clifford's Phonics Fun readers. She published Scholastic's first basic reading program, Literacy Place. More recently, she was involved in the development of READ 180, an intervention program for students in grade 4 and above, and Scholastic Red, a professional development program. Ms. Alexander is the former Deputy Assistant Secretary for Policy and Planning in the Office of Educational Research and Improvement, U.S. Department of Education; Executive Director of the National Council on Education Standards and Testing; Deputy Director for Curriculum with The Edison Project; Associate Superintendent of Public Instruction in California. She is also on the Advisory Board for Advantage, Inc.

    *David J. Alukonis, Chairman, Hudson School Board, Hudson, New Hampshire, holds a B. S. in business management.

    *Amanda P. Avallone, Assistant Principal and Eighth-Grade Teacher, Summit Middle School, Boulder, Colorado, has a BA in English. Summit, named an NCLB Blue Ribbon school, is a charter school with an enrollment of 88% white, 10% Asian, with about 28 percent of students identified as gifted, the highest percentage of gifted students in the Boulder Valley School District. Summit Assistant Principal Amanda Avallone, who also teaches the school's highest level English class, said the students "keep me on my toes."

    Her writing experience includes creating educational materials and curriculum guides for Turner Broadcasting and CNN programs.

    Honorable Jeb Bush, Governor of Florida. Bush attended Phillips Andover, a private boarding school in Massachusetts. He has a BA in Latin American studies from the University of Texas, Austin. After moving to Florida he worked as a business property realtor and was involved in various entrepreneurial pursuits, including working for a mobile phone company, serving on the board of a Norwegian-owned company that sold fire equipment to the Alaska oil pipeline, becoming a minority owner of the Jacksonville Jaguars, buying a shoe company that sold footwear in Panama, and getting involved in a scheme to sell water pumps in Nigeria. Bush co-founded the first charter school in Florida, Liberty City Charter School, a K-6 elementary school.


    Bush was awarded three Rotten Apples in Education Awards 2003 from Phi Delta Kappan researcher Gerald Bracey: the "There are already too many old people in Florida" award; the "Because piglets are cuter than kids" award; the "Parents don't need to know what their kids are tested on" award.


    Intrepid researchers might ask the governor what qualities he looks for in naming someone chief reading officer of Florida. Interesting connections worthy of the steamiest tabloids.

    Barbara Byrd-Bennett, Chief Executive Officer, Cleveland Municipal School District Cleveland, Ohio, has a M.S. Educational Administration. Educated in New York City, she was Supervising Superintendent for the Chancellor's District, responsible for direct oversight of the lowest performing schools in the New York City public school system. She is on the board of directors of the Albert Shanker Institute, whose other members include E. D. Hirsch, Diane Ravitch, and Stanley Litgow, Lou Gerstner's IBM cohort at the education summits. Cool Cleveland provides an opinion on her corporate connections:


    *Carl A. Cohn, Superintendent, San Diego City Schools. Cohn has an Ed.D. in Administrative and Policy Studies. Under his superintendency, the Long Beach Unified School District became the first public school system in the U.S. to require school uniforms for students in grades K-8. Cohn's motto was "High Standards: Dress, Behavior and Achievement." He favors deregulation of teaching licensure. "To me, that makes no sense at all, if the youngster graduated from an Ivy league school and really knows English literature, let them go teach it."


    In 2001, Cohn received the Harold W. McGraw Jr. Prize in Education for his leadership in boosting student achievement. Writing in the Nation, Stephen Metcalf pointed to the high number of recipients who used or promoted Open Court, including Mr. Cohn. Cohn is advisor for the Broad Superintendents Academy, serves on the boards of the Re-Inventing Schools Coalition, the Wallace Foundation Education Leadership Advisory Committee and the National Heritage Academy. He is president of Urban School Imagineers, an educational consulting firm.

    *Shirley V. Dickson, Educational Consultant, Laguna Niguel, California, has a Ph.D. University of Oregon in Special Education/Reading, the home of direct instruction. She is one of 72 Reading First review panelists tasked with reviewing state applications for $900 million in new Reading First grants. In the summer of 2004, she went on the road with Teacher-to-Teacher, a U. S. Department of Education program offering "research-based practices and effective methods of using data to inform instruction." Dickson is the coauthor of a paper for the Council for Learning Disabilities promoting NCLB.


    John Q. Easton, Executive Director, Consortium on Chicago School Research, holds a Ph.D. in Measurement, Evaluation, and Statistical Analysis. One of his reports for the Consortium, on the district's retention policy, concludes, "There have been impressive increases in the proportion of students who meet the test-score cutoff for promotions."


    For a look at the special relationship between the Chicago Public Schools and the Consortium, including a $100,000 consulting job for Easton, search the Substance archives.

    Honorable Dwight Evans, Member, Pennsylvania House of Representatives, has a B. A. in English. The leaders of five private educational management organizations (EMOs) now running Philadelphia schools donated to his political campaign.


    Evans was honored for his work promoting the Philadelphia hospitality industry. In 2004, he co-taught a political science class at Temple University, "Public Policy and the Black Community," a class limited to 16 students. In his run for Philadelphia mayor, Evans advocated charter schools and vouchers.

    David W. Gordon, Sacramento County Superintendent of Schools, received an Ed.M. in education administration from Harvard University. He is on the U. S. Department of Education speaker's circuit, advocating streamlined admission for second career teachers. At the California State Department of Education Gordon was director of Program Evaluation and Research and had lead responsibility for developing California's student proficiency tests and the CBEST Teacher Competency Test. He is on the editorial board of the Peabody Journal of Education and was appointed by George W. Bush to serve on the newly-created President's Commission on Excellence in Special Education. Here's what he told Ed.magazine, the Harvard Graduate School of Education publication: "There is a growing movement in our country that testing is useless or even harmful. Boycotts of testing have sprung up in many upper middle class enclaves. I understand the frustration of those parents about overtesting, but we are one nation--and we have many young people who are only stepping up to the door of opportunity. Our poor and minority students will have to learn to do well on tests to help open those doors. We can reform our testing system in the long term, but for now we owe it to all children to help them meet the standards that will allow them to compete on the same terms as more advantaged students."

    Gordon has served as an Associate in Education at Harvard University, a visiting scholar at Stanford University, and a visiting professor at the University of California, Riverside, and as a speaker at the Milken Institute Global Conference.

    *Kathi M. King, Twelfth-Grade Teacher, Messalonskee High School, school number 940 in Oakland, Maine. She has an MA in mathematics and teaches advanced placement calculus. Her bio as adjunct instructor at Thomas College Institute indicates "King directs one of the most progressive high school mathematics departments in Maine."

    Honorable Keith King, Member, Colorado House of Representatives, holds an M.S. in Vocational Education. King started Waterbed Palace in 1977 and sold it in 2002 after opening 18 retail stores around the nation. Named Retailer of the Year, 1991, King was President of the Cheyenne Mountain Charter Academy 95-98. Cheyenne Mountain attribute their success, from the start, to implementation of Reading Mastery.


    According to the Center for Public Integrity, King is also a director of Christian Women's Club, Colorado League of Charter Schools, Cmca Foundation, Eagle Feeders, LLC, Combs, Schachterle, King, King.

    In a questionnaire, King offered these comments:
  • Favorite president and why: Reagan - He was able to communicate with people.
  • If there were one law you could pass or repeal in 2005, what would it be? I would love to pass a bill that funded school districts based partically [sic] on the academic achievement of students.
  • "I support the family unit of one man and one woman as the most important institution of our society."

    *Kim Kozbial-Hess, Fourth-Grade Teacher, Fall-Meyer Elementary School, a K-6 school numbering 240 students in Toledo, Ohio, has an MA in education. King's Internet project on the Underground Railroad received a Pioneering Partners Award. She helped develop Ohio's social studies content standards. As a member of FairTest's Assessment Reform Network, she made this post: "As a teacher in Toledo (Ohio) Public Schools, I am very proud of our union. Our Intern/Intervention Program has gained national recognition using highly successful veteran teachers to work with new teachers and teachers with difficulties."

    Andrew C. Porter, Director, Learning Sciences Institute, Peabody College, Vanderbilt University, holds a Ph.D in Educational Psychology. His current work focuses on curriculum policies and their effects on opportunity to learn. Currently, he has research support from the National Science Foundation; co-director, System-Wide Change for All Learners and Educators; principal investigator, Longitudinal Design to Measure Effects of MSP Professional Development in Improving Quality of Instruction and Science Education; principal investigator, Improving Effectiveness of Instruction in Mathematics and Science With Data on Enacted Curriculum; ED's Institute for Education Sciences (Consortium for Policy Research in Education). He is an elected member and former officer of the National Academy of Education, Lifetime National Associate of the National Academies, and past-President of the American Educational Research Association.

    Luis A. Ramos, Community Relations Manager, PPL Susquehanna, holds a BS in Management Science. He is a founding member of the Hispanic Business Council, a wing of the Greater Lehigh Valley Chamber of Commerce; member Pennsylvania State Board of Education; appointed by U.S. Sen. Rick Santorum to the U.S. Senate Republican Task Force on Hispanic Affairs; trustee DeSales University; member Allentown School District High School Reform Oversight Committee.

    Mark D. Reckase, Professor, Measurement and Quantitative Methods, Michigan State University, has a Ph.D. in psychology. Prior to that he was the Assistant Vice President for Assessment Innovations at ACT, Inc. Reckase headed a team of teachers and assessment specialists who designed American College Testing's (ACT) portfolio assessment program called PASSPORT.

    Of interest is a paper, "The Controversy over the National Assessment Governing Board Standards," Brookings Papers on Education Policy 2001. Here is an excerpt:
    This paper provides an analysis of the controversy surrounding the standard setting process conducted by ACT Inc. for the National Assessment Governing Board (NAGB). 1 This process is the most thoroughly planned, carefully executed, exhaustively evaluated, completely documented, and most visible of any standard setting process of which I am aware. Extensive research was conducted to determine how best to develop each step in the process. 2 A distinguished team of experts guided the process through its development and implementation. 3 And, the process has been open to scrutiny with evaluators observing the design and implementation of every step. . . .
    In June 2005, Reckase presented "Policy Implications for Value added assessments" at the National Association of State Boards of Education Study Group on Value Added Assessment.

    John H. Stevens, Executive Director, Texas Business and Education Coalition, received a BS in English and did graduate work in psychology. Rod Paige appointed Stevens to be the representative of the American business community on the Negotiated Rule-making Panel for the implementation of No Child Left Behind and the Internal Review Panel for Title One. At a July 2000 education reform conference, Stevens said, "Educators do not dominate the dialogue on education in Texas. For more than a decade, the business community and a group of key legislative leaders ... have been the major players in shaping state education policy."

    In the Business Roundtable toolkit for business leaders, "Using the No Child Left Behind Act to Improve Schools in Your State," Stevens is the Texas contact. http://tinyurl.com/8zpmo

    When Rod Paige was named U. S. Secretary of Education, Stevens offered this appraisal, "He'll use his bully pulpit to create a sense of seriousness. He'll expect schools to succeed without making excuses. He's a stickler for accountability." Linda McNeil's research poignantly describes the corporatized numerical definition of success offered by Stevens.

    Mary Frances Taymans, Executive Director, National Catholic Educational Association Washington, D.C., has an Ed.D in Curriculum and Instruction. At the National Catholic Educational Association she works specifically in the areas of leadership and faith development.

    *Oscar A. Troncoso, Principal, Socorro High School, Socorro Independent School District, El Paso, Texas, has an MA as an instructional specialist.

    Honorable Thomas J. Vilsack, Governor of Iowa, has a law degree. Governor Vilsack is the immediate past chair of the Democratic Governors' Association and a member of the National Governors' Association Executive Committee. He is the former Chair of the national Jobs for America's Graduates (JAG) program, a founding member and former Chair of the Governors Biotechnology Partnership, the former chair of the Ethanol Coalition, and the former chair of the Midwest Governor's Conference. He is chair of the Democratic Leadership Council, of which Hilary Clinton is chair of the American Dream Initiative. Here is an education proverb from Governor Vilsack: "When I read Thomas Friedman's book, The World is Flat, I was struck by the African proverb he mentions and reflects on in that book. In it, he basically says that every morning in Africa, a gazelle wakes up, and recognizes that it must run faster than the fastest lion or it will be killed. And every day in Africa, a lion wakes up, and recognizes that it must run faster than the slowest gazelle or it will starve to death. It doesn't matter whether you are a lion or a gazelle, when the sun comes up, you better start running.

    "This fall, over 40,000 Iowan kindergarteners will enter school and begin their first steps toward graduation in 2018. These children will undoubtedly be their families' providers, as well as our nation�s leaders and workforce. Those kindergartners, as well as students of every age throughout our country, will need to run faster."

    Michael E. Ward, Former State Superintendent of Public Instruction, North Carolina Public Schools, holds an Ed.D in Education Administration & Supervision. He has been president of the Council of Chief State School Officers, a teacher, and a principal.

    Eileen L. Weiser, Member, State Board of Education, Michigan Department of Education., has an M.M. in piano performance. Her husband is U.S Ambassador to Slovakia. Since his arrival in 2001, he has been extremely active, having officially visited all 138 cities, some more than once, and numerous villages. As documented by the Center for Responsive Politics, Weiser, the founder and CEO of real estate investment company McKinley Associates Inc., was the Michigan finance chairman for the Bush campaign and one of the Bush Pioneers, whose members each raised $100,000 or more for Bush. Weiser contributed $24,900 to Republican candidates and party committees in 1999-2000, including $2,000 directly to the Bush campaign. He also contributed $5,000 to Bush's Florida recount effort. Weiser's wife, Eileen, contributed $15,350 to the GOP, including $1,000 to the Bush campaign McKinley & Associates was responsible for $287,750 in individual, PAC, and soft money contributions during the 1999-2000 election cycle, all to Republicans.


    Ex-officio Member Grover (Russ) Whitehurst, Director, Institute of Education Sciences, U.S. Department of Education., he holds a Ph.D. in experimental child psychology . Formerly Leading Professor of Psychology and Pediatrics and Chairman of the Department of Psychology at the State University of New York at Stony Brook.

    Acting on President Bush's behalf, Reid Lyon recruited Grover Whitehurst, formerly chair of the psychology department and a professor of pediatrics at the State University of New York at Stony Brook, to head the department's Institute of Education Sciences, which has become the administration's bully pulpit for encouraging science-backed teaching methods in all subject areas. http://www.city-journal.org/html/14_3_education_pres.html

    Grover Whitehurst announced the release of 2005 NAEP scores, explaining that "Achievement levels were developed by the National Assessment Governing Board, or NAGB, to set standards for what students should know and be able to do. For each subject and each grade, NAGB has established standards for Basic, Proficient, and Advanced performance. When we make comparisons using achievement levels, we usually look at the percentage of students who performed at or above Basic, which also includes students performing at the Proficient and Advanced achievement levels, and the percentage who performed at or above Proficient."


    Gerald Bracey points out that the word "proficient" is a trap. Not many children reach it on NAEP. Although NAEP achievement levels have been rejected by anybody who has ever studied them, media headlines march on. The problem is that NAEP levels don't agree with anybody else's levels. As Lyle Jones pointed out: American fourth-graders were third in the world in TIMSS science, yet only 12% were at the proficient level and one percent at the advanced level in the 1996 NAEP science assessment. Similarly, Bracey has observed that on the 2000 NAEP reading assessment, only 32% of fourth-graders attained proficient or better, but that American 9-year-olds were second in the world among 27 countries in the IEA reading study, How in the World Do Students Read? In the 2003 math assessment, American fourth-graders were well above average on the TIMSS math test and third in the world in science. But only about a third showed up as proficient on NAEP math and science test administered the same year. Kids who are virtually on top of the world are not proficient? It makes no sense. . . unless the people governing NAEP have another agenda. But we have a media who write from press releases, unions who look the other way, and Colleges of Education that don't seem to care.

    Under NCLB, NAEP becomes mandatory: All states must participate. There's no reason to do this other than, as Bracey concludes, "to use the NAEP achievement levels except to beat schools over the head . . . . Critics will take the discrepancy between the state results and the NAEP results as evidence that the schools are still failing and that the states are lying to their citizens." Writing in Dissent, Fall 2004, Bracey notes, "And it gets worse. The NAEP levels are not only ridiculously high, they are 'fundamentally flawed,' to use the words of the National Academy of Sciences. The NAEP achievement levels have been examined and found wanting by the National Center for Research on Evaluation, Standards, and Student Testing; the National Academy of Education; the National Academy of Sciences; the General Accounting Office; and individual psychometricians. The reports say that the process is confusing, internally inconsistent, and lacking in evidence for validity. These conclusions would condemn any proposed commercial test to the trash bin. But NAEP chugs along ignoring the flaws. Having many students score low has political uses."

    FairTest: the Center for Fair and Open Testing weighs in on NAEP:

    THE CLAIM: U.S. Education Secretary Rod Paige said that NCLB only asks "that a third-grade child read at a third-grade level," and defines "proficiency" as basic grade-level capability.

    THE REALITY: The National Assessment of Educational Progress (NAEP) proficiency standards, on which NCLB is based, are set so high that it will be impossible for most schools to reach them.

    "Multiple independent studies of how NAEP set proficiency levels found that the procedures were flawed, resulting in absurdly high cut-off points. In other words, "proficient" on NAEP means a score well above what is typically considered "grade level."

    A federal Government Accountability Office study found that states vary wildly in how they define proficiency. "Proficient" is far easier to reach in some states than in others. The differing percentage of students scoring proficient therefore does not reflect the relative academic health of their schools and students. http://www.fairtest.org/nattest/Reality_Testing_NCLB.html

    *Members of Assessment Development Committee

    — Susan Ohanian
    Substance, November 2005


    This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.