The facts vs. David Brooks: Startling inaccuracies raise questions about his latest book
Factual discrepancies in the New York Times columnist's new book raise some alarming questions about his research & methods.
This seemed like a great stat to consider for my book, so I set about doing what I always aim to do when I come across a secondary source: I checked the facts. Before segueing to my current career as a writer, I spent about six or seven years as a magazine fact-checker, mostly for CondÃƒÂ© Nast magazines; seeking primary sources for facts is my default mode.
I began by emailing a media contact at Gallup. After a series of exchanges, I was relayed to another Gallup employee. Ultimately I communicated directly with three different people at Gallup, and they communicated with multiple other staff, including a group in Princeton who combed Gallup's print archives. During the process, a Gallup employee and I also emailed Brooks, but neither of us got a response. (Granted, I didn't expect one, given that I had emailed his New York Times address, which likely receives a torrent of messages daily. I assume Brooks has another, more private email address he actually uses.) In their last email on the matter, the representatives at Gallup stated their final verdict: They did not "have any record of the data" that Brooks cited. But they also suggested that I try contacting Brooks again, and that if he shared any new information they would reopen the inquiry.
It was frustrating that they wouldn't definitively just say Gallup didn't conduct the polls Brooks cited, but if a number of people on the staff couldnÃ¢€™t find the polls then I knew either Gallup had atrocious record keeping, or Brooks was wrong. I hated that this tiny window of doubt was open, but there didnÃ¢€™t seem to be a path to find out one way or the other, so I gave up and didn't include the passage in my book.
Fast forward to April 2015, when I came across a review of The Road to Character in the New York Times Book Review. To my amazement, the same statistic from the Aspen talk, once again credited to Gallup, was paraphrased. And my obsession with the mysterious Gallup poll roared back to life.
It was one thing if Brooks made erroneous claims in a one-off lecture. It was something else entirely to put those same claims in a book. Now I had to get to the bottom of this. I wanted to fact-check the review to make sure the reviewer was accurately quoting from the book. I did a search on Google Books for part of the quote and found it. After that, I went to Barnes and Noble and looked up the passage in the actual book just to confirm it was there.
The passage from The Road to Character reads:
Over the course of my search I discovered other iterations of the passage as well: During a 2011 appearance on "Real Time With Bill Maher," for example, Brooks tells the same exact story, except in this telling, the second study took place not in 2005 or 2006, but in 1998.
The NYT's review of The Road to Character itself picked up the pattern of changing dates, when the reviewer noted that the passage in question was similar to one in an earlier Brooks book, The Social Animal, only it was written "with slightly different dates."
The passage from The Social Animal reads:
Somehow, between the publication of The Social Animal in 2011 and the publication of The Road to Character in 2015, a study that originally occurred, by Brooks/ telling, in "the late 1980s" became one that occurred nearly 20 years later. (Amazingly, to the New York Times reviewer, the late 1980s and 2005 are only "slightly different dates." And how was any difference in dates for the same citation, no matter how "slight," not problematic to the Times reviewer?)
What began as a simple fact-check of a Gallup poll was devolving into a morass.
I contacted Brooks' publicist at Random House, and found the email address for a student at Yale who was thanked in the acknowledgments of The Road to Character as his fact-checker. The Yalie never replied, but I did get a response from his publicist, who told me that she had to check with Brooks, and that she would get back to me. It took over a week, but she replied with an admission that the study was wrongly ascribed to Gallup, but that the "sociological trend is accurate." She also provided a link to what she said was the actual study ( Changes in Adolescent Response Patterns on the MMPI/MMPI-A Across Four Decades, an analysis by university scholars of various data sets), and said the passage would be corrected in future printings of the book.
When I clicked the link and read the abstract, the first thing I noticed was the paper itself was published in 2003. The Road to Character stated that the data set was from 2005; how could that be when the final published paper came out two years earlier? That paper listed two relevant sets of data: One set collected in 1948 and 1954, and another set from 1989. I emailed Brooks' publicist again. Had he merged the 1948 and 1954 polls into one that he then listed as having occurred in "1950"? And what the hell was up with the 2005 date listed in The Road to Character?
I did not receive a reply.
I then contacted two of the authors of the paper, Cassandra Newsom, a professor of pediatrics, psychiatry, & psychology at Vanderbilt University, and Robert Archer, a professor of psychiatry at Eastern Virginia Medical School. Abstracts and studies in general can be a challenge for the layperson to read, as academic text is often as opaque as legalese. So they sent me the actual paper and I went through it over the phone with Newsom. Here's where things really began to unravel.
Newsom explained that, indeed, the first data set was from samples taken in 1948 and 1954, not 1950, and the second data set was from 1989, not 2005. Also, the first data set was exclusive to Minnesota, where respondents were disproportionately white, while the second was a more heterogeneous national sample. The respondents were not high school seniors, as Brooks wrote: In the '48 and '54 set, respondents were ninth graders; the 1989 sample was composed of 14- to 16-year-olds. (The '48 and '54 data set was later sifted to match the 1989 set by including only ninth graders who were aged 14-16.) Lastly, in the 1989 set, 80 percent of boys answered "true" to the statement "I am an important person," and 77 percent of girls said true. Yet Brooks cited 80 percent as the only figure. I asked Newsom if it was possible that some other study was done in 2005 with the same "I am an important person" question. She said there has not been another normative sample since 1989.
(I emailed Brooks' publicist a third time, asking her to contact Brooks for clarification on the stream of additional apparent inaccuracies but never heard back.)
The thing I keep wondering is how did Brooks get nearly every detail of this passage wrong? He said Gallup did the polls, when they were actually done by academics. He merged a data set from 1948 and 1954 into 1950. He said the second data set was from 2005, when it was from 1989 (to me, the most damning and damaging inaccuracy). He said it was high school seniors, when it was ninth graders. And he said 80 percent answered true, when that was only so for boys. Can one accidentally get this many details wrong?
So the question is, if it wasn't an accident, why would Brooks deliberately falsify nearly every detail in a passage of his book, let alone one that is a cornerstone of the book's P.R. campaign? If you Google David Brooks and "Gallup" a vast scroll of media mentions of this erroneous passage appears. Brooks is a very talented writer, and whether you agree with him or not, he's highly skilled at expressing ideas in compressed, digestible bites. After all, that is his job as an Op-Ed columnist. As I dug into the case, I watched the Aspen video again, while also referencing other lectures Brooks gave. Every time, he spoke with pith and humor; the audiences laughed and cheered along with his jokes and likably nebbishy demeanor. The guy knows what works.
As his publicist suggested, he may be right about the general trend. So why did he feel the need to gild the lily? Why couldn't he have referenced the paper using the correct statistics? Perhaps it is a sign of his skill as a communicator, and his weakness, to know how to make a passage particularly seductive. On "Morning Joe," as one of the hosts recited the erroneous passage, another host waved a cellphone, signaling, "See? This is the problem with today's youth!" Saying 1989 for the latter date wouldn't quite work for that bit.
In addition to his factual errors, it's worth noting that Newsom and Archer challenge Brooks' interpretation of their paper. Newsom explained to me that "I am an important person" was one question in a subset of questions related to "Ego inflation." Interestingly, though this one question had a huge jump, ostensibly supporting the case of less humility over time, in fact, the overall subset that this question was a part of -- Ego Inflation -- had a relatively small increase from the first data set to the second. ItÃ¢€™s generally not sound to spotlight one question in isolation, especially if it contrasts with the findings of the overall study or subset.
One reason is language itself. The words "I am an important person" may have been interpreted differently in the '40s and '50s, than in 1989. Archer said that the early sample may have interpreted the item as meaning "more important than others" whereas the '80s sample may have read it to mean "I believe I am a valuable person." Further, though the authors compare a sample from Minnesotans to a national sample, it's done so in a paper intended to be read and interpreted by academics, who can weigh the discrepancy. Mainstream readers would likely benefit from knowing the samples are not apples to apples, as Brooks' citation implies.
As critical as the point of this paragraph and the previous one is, interpreting data is highly complex and not always an exact science and one could argue over what's reasonable to include or exclude when referencing academic papers for a general audience; the focus of this essay is on factual inaccuracies. (But considering my intense spotlight on this passage, I'd be remiss to not at least mention the paper's authors' displeasure with Brooks' cherry-picking a question out of context, drawing a misleading conclusion from it, and it becoming the lead sound bite in the media for a No. 1 bestselling book.)
* * *
It's generally considered reasonable by fact-checkers and editors to allow for quotes from different sections of an interview to be merged, as long as the meaning behind them isn't misrepresented. When you read quotes in an article, they've often been cleaned up for grammar, and the "ums" and tongue clucking that populate regular speech are generally deleted. But those changes are made with the intent of bringing clarity. Statistics, especially ones from scientific papers, need to stay precise and accurate. Changing them will almost always be construed as done in the name of persuasion. It's a dangerous ploy to start fudging numbers and wordings from academic studies.
How did so many details from this passage change from one book to the next? How did the late 1980s morph into 2005, teenagers to high school seniors, and a personality test to the Gallup Organization? It's interesting that as the passage distanced itself further from the truth in its second iteration, it also became more specific in its details.
I then watched the Aspen talk again and remembered something. I had another statistic from Brooks in my research notes for Invisibles. In the talk he said: "In 2006, 51 percent of 25 year-olds said that being famous was the most important life goal they could have." In The Road to Character, the text is massaged to: "By 2007, 51 percent of young people reported that being famous was one of their top personal goals."
The endnote in his book lists "How Young People View Their Lives, Futures and Politics: A Portrait of 'Generation Next'; The Pew Research Center for the People & the Press" as the source. But the Pew survey itself reads: "When asked not about themselves but about their generation, most 18-25 year-olds say getting rich and being famous are important goals for people in their age group." [Emphasis mine.]
This is a nuanced but critical difference. Young people were giving their impression of their generation, not observations of themselves personally. How we view our culture or a subset that we are a part of is, of course, not necessarily the same as how we view ourselves. Again, like the changes he made to the "I am an important person" passage, it's worth noting that Brooks' revision of the language here, too, results in a more persuasive sound bite.
What makes the case against mere catastrophic sloppiness is that, oddly, Brooks had the luxury of a fact-checker. Unfortunately, due to the expense, it's rare for writers to have dedicated fact-checkers outside of high-end magazine journalism. Can every New York Times book reviewer be expected to fact-check any content they reprint or paraphrase? (Maybe not. But if you know a passage citing statistics is a repeat from an earlier book, but with different dates, probably, yeah.) What about TV show hosts? Even when writers have the best intentions, it is inevitable that errors will be introduced into their articles, and even more so books that are hundreds of pages long. We must ask, what errors, to a reader, are acceptable?
In a recent Atlantic article the writer lamented that if there isn't room today for a writer like Joe Mitchell -- a New Yorker scribe of yore who fabricated significant passages in his journalism -- "The problem isn't him--it's us." But facts always matter. And if the writer is practicing some form of "new" journalism, then he should let the reader know, perhaps the way films use the "inspired by" disclaimer.
* * *
The question I keep wondering, and what I think perhaps is most relevant with Brooks, is why? Why would someone with this level of prestige and influence be so woefully sloppy in his reportage -- or worse? Imagine yourself for a moment as an Op-Ed writer for the most influential newspaper in the world; you get paid huge sums of money for a string of bestselling books, you entertain and enlighten live crowds at your lectures, you get paid to spout your opinions on TV. I don't know what it would do to my head if I had the level of influence that Brooks has as a writer and cultural commentator. Perhaps, as a friend of mine suggested, maybe Brooks is just glib.
Ultimately, maybe it doesn't even matter whether the passage was fudged on purpose or not. Perhaps to be so careless shows the same degree of culpability, and condescension toward the reader, that willful manipulation does.
As I was about to send the draft of this article to my editor I did a quick review again of The Road to Character on Google Books. To Brooks' publicist's word, based on my email to her, future printings of the book, indeed, appear to be different. Now the passage reads:
I was astonished. Even this "correction" isn't correct! Where is 1954? Where's the 1989? In the interest of fact-checking, and a humility I assume Brooks would approve of, I should note, it is possible that a study was done in 2003 that had 80 percent of all respondents saying, "Yes, itÃ¢€™s true, I am an important person." I'm still trying to find out.
David Zweig is the author of Invisibles: Celebrating the Unsung Heroes of the Workplace.
FAIR USE NOTICE
This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of education issues vital to a democracy. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information click here. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.