See also The Wall  Street Journal, The Evidence of Things Not Seen, Friday, May 16, 2003.

by CHETLY ZARKO
ANN ARBOR, MI
Central to each of the defenses against my critiques ofPatricia Gurin's expert testimony (laid out by Julie Peterson, Paul Courant, Geral Gurin, John Matlock, and the U Record) is the claim that the "Michigan Student Study May 24, 1994 Executive Summary" was "preliminary" and based upon only the first two years of data from a study designed to analyze four years. They also claim that the later data analysis was "methodologically superior."
Below I provide 5 samples (photos of original documents) of raw four year trend data where statistically significant regressions find decreasing perceptions of interracial friendship over time, increasing racial tension, and other negative indicators that contravene Gurin's claim of "remarkable consistency" in the data.
What's worse, Gurin's expert testimony did not cite key portions of these facts or other data from her dataset that would oppose her testimony.  Normally, one would expect, as Gurin's defenders suggest, that later studies would, due to the extra time available and experience with the dataset, use better methodology.  In these cases though, when NEGATIVE FOUR YEAR TREND DATA WAS AVAILABLE, Patricia Gurin or the authors of the "new" 1998 Internet version of the Michigan Student Student report often chose to only report ONE-YEAR SNAPSHOTS citing racial differences (or non-differences) without noting the importance of the trend changes. Perhaps the word-choices (secret, hidden, misleading, lying) in my own "preliminary" analyses were simply too weak, but then again, one should never assign to intent what could be assigned to incompetence or the researchers own hidden biases (and in this, I grant the possibility that Dr. Gurin was not ill-motivated). As with most "good science," she could have included the sharply negative data and suggested alternative hypotheses still fitting U-M's legal position (it would have been harder and required more creativity), rather than choosing to "refocus" on only the "positive" areas of the dataset.  As it was, Gurin chose to tell the court that the data exhibited a "pattern of consistency" that was "unusual" in her social science experience. It was "unusual" that it was that consistent because it wasn't that consistent. Gurin's insistence that the data and results are so highly consistent with exactly one possible policy is the truly inconsistent result. Good science however isn't necessarily what a lawyer needs in justifying an almost impossible to justify set of behaviors.
Sorting through the many (confusing) University of Michigan responses to my concerns can be a difficult process, and this analysis is not an attempt at a "comprehensive counter-response" to each response U-M levied against my various work on this matter.  I simply present here a renewed case to question the accuracy and integrity of the scientific work performed on the Michigan Student Study.
A scientific analysis of the over 160 documents I found in the same folder with the 1994 summary was conducted by Dr. Robert Lerner and Dr. Althea Nagai. There work supports my conclusions and is available at the Center for Equal Opportunity's website.  At the time I wrote each of my analyses, and especially when I wrote the Wall St. Journal piece, I was aware of these supporting documents and generally aware of what to conclude from them, but I was still awaiting the final scientific analysis to be completed. I was unaware however that Dr. Lerner would also discover that the questionnaires had been changed in the final study year (20% fewer questions overall, a key battery of questions having four gradations of response instead of five, and a key question undergoing substantive content change).
The samples I provide, although not exhaustive, are reflective and the most salient examples of data that is not "consistent" with Dr. Gurin's claims. Remember also that the remainder of the dataset has not been released (at the time of writing) and that Michigan continues (it disclosed the raw data pursuant to a FOIA lawsuit two years later) to refuse to disclose them as required by FOIA.
Like clockwork, each diversity defender cites the second page of the nine page report, which notes its own weaknesses. Most "good science" notes its own weaknesses, unlike Patricia Gurin's expert testimony, which found 'remarkable' "consistency" in the results leading to her conclusions. Gurin's 1999 testimony sets a high-bar when she says the data reflected a "pattern of consistency." It is against this back-drop that one can find the stark contrast to her husband's work which presents a "mixed bag" of conclusions favoring both sides on some issues, favoring neither on others, and neutral on many. The 1994 summary also notes the fact that the body of the text was written before the fourth year data was obtained, but literally as they were writing, the final touches were being put on processing the data collected for the last sampling.  The authors were clearly aware of this data, and much of its content. They did nothing to modify key parts of the report that they would have allegedly changed they had only "had all the data."  They had all the data; they simply hadn't processed it as fully as one would have liked.  This leads us to the question of why it took four years before a final version of the summary was published to the Internet in 1998 (the data processing was being finished in early 1994). Four years is a curious amount of time to issue a "final report" (not necessarily to do a whole new piece of science though, which is what Patricia Gurin claims to have done from 1997 to early 1999). The authors must have struggled with how to present this material safely to the full public, especially with Prop 209 (1995), Hopwood (1996), and the the filing in 1997 of the now-historic lawsuits. I would propose that the odd four-year lag backs the theory that the 1994 report was intentionally relegated to the ashbin of history (the fact that U-M was smart enough to include it in discovery as Exhibit 17,529 is irrelevant since the plaintiff attorneys only represent two individuals).  The 1998 Internet publication mirrors its partner, the 1999 expert testimony, in how it obtrusely ignores the trend data at exactly the most important moments.
The 1994 summary was prepared for a retreat held by the U-M "Council on a Multicultural University." Julie Peterson also claims that the numbers of people in this group, and a select few outside of it, who received the report prove it was not "secret" or "hidden." At that time, with no lawsuit pending, the importance of the report was unknown and none of the limited few receiving it either had motive to disclose it or access to the supporting documents. Even if it wasn't fair to contemporaneously label a report "hidden" after it was found in a "restricted" part of an historical archive following a FOIA request, the circle of people receiving this information were unlikely to disclose anything.  All of these policymakers were expected to be discreet, and the recommendations of the summary itself read in a way that supports the idea that the summary wasn't a widely circulated document. Gurin and Matlock wrote the report well to their bureuacratic purposes (as both "good science" and "to cover themselves," they emphasized the weaknesses and future needs/possibilities for research). Indeed, many of the possibilities suggested for future work, like a detailed investigation of the linkages between socio-economic status, race, grade points, test scores, and college performance, were never followed up upon (at least not in the 8 dissertations the MSS dataset since 1994 or the known public documents about it). Such a follow-up, on the socio-economic angle (the MSS contained all the data necessary to do a nice comparison), could naturally implicate whether U-M's race policy overreaches when instead an economic policy would be more targeted. For those of us interested in socio-economic preferences, we are left only tantalized - and without the data or a person trumpeting the cause to properly debate the issue. Of course, Michigan's legal defense requires that there can be no other "more narrowly tailored" alternatives, hence they must deny or prevent such claims.  Rather than look for a potentially embarrasing answer, of course, it was easier for U-M to simply deny, without scientific evidence, the claims that a race-neutral scheme could be devised that would still benefit blacks while helping all "disadvantage" individuals.  If only they would have tried the slightly more intellectually honest defense, of saying they simply didn't look; but then again, the case wouldn't have made it this far otherwise.
A final, consistent theme of the "response" to my concerns is the priceless argument that "perceptions are not experiences." [Courant, Peterson, Gerald Gurin, and Matlock] This is priceless because, naturally, experiences and perceptions go hand-in-hand; you can't have one without the other.  The basic argument here is that the 1994 Executive Summary focused on parts of the Michigan Student Study related to student perceptions of racial climate whereas the later two studies focused on measurements of the "actual diversity experience."  The problems with this claim are that the Executive Summary contains valid criticisms related to individual "experiences", and the later work also uses "student perceptions" to measure experience (of course, how else do we measure experience).  The experience/perception dichotomy is a false dichotomy constructed specially as a response to my work (Gurin's expert testimony doesn't inform us of this nuanced pitfall because she succumbs to it herself).
A best formulation of this argument is the most recent, found in the The University Record.  Nicely, it quotes Patricia Gurin (therefore her first response to this issue): "According to Gurin, students' perceptions of racial tension on campus 'do not tell an accurate story about their own relationships.' In her expert report, she notes that in their senior year students reported very little tension or hostility in their own interracial friendships at Michigan."
Presented this way, it could (and really should) be interpreted as an individual "self-perception" versus individual "perception of others" dichotomy.  This is very familiar to political science surveyors who have debated for years whether individuals vote on their own "personal condition" or the condition of the nation as a whole as they perceive it (my understanding is that it is the latter, and nobler, view that wins out, on average). In pyschology, I would suggest that BOTH MEASURES are important, because as the chair of a psychology department must know, "attribution bias" plays havoc with "self-reported" indicators.  Attribution bias is the pychological principle that an individual is more likely to (subconsciously) attribute "bad motivation" and behavior to others and less likely to attribute it to themselves. This causes a bias in favor of the "self-reporting" individual and against perceptions of the group.  It is such a fundamental principle in psychology that I keyworded the entire online Gurin expert testimony and never even found the word "attribution," except for twice in reference to referencing items. She either makes no effort to "control" for this obvious effect, or the results after such a control are intentionally left out.  We have a failure to control for "attribution bias," or at least mention it as a study weakness (and mention the converse data showing "environmental" perceptions of tension rise).  What we have really is a study of all different kinds of perceptions, most of which are simply unreliable because, simply ...  they are perceptions. Even if "unintentional," that U-M tried to use unwashed perceptions to influence a Court is inexecusable (at least without being forthright about their weaknesses, and not by portraying the data as having a "remarkable pattern of consistency").
So now we have college seniors reporting little of their "own" hostility and more of the hostility of "others" in the "environment."  Should we give more weight to one side, or the other (or perhaps ... both)?  Patricia Gurin's "expert" testimony is left exposed; each of U-M's many responses all suggest that self-reported "personal experience" trumps "personal perceptions of the environment" in calculating whether "diversity works" (increases "democracy" or "learning outcomes"). This distinction though is also only a post-hoc contruction in response to my Wall Street Journal analysis.  This is illustrated by the fact that even as Patricia Gurin decided only to give weight to the attributions most beneficial to her view, she still tried to use it to prove the very point she criticizes me for making. Quoting her testimony, "The proportion of white students who had at least one close friend of color (among their six best friends) increased from about one third (32%) at the time they entered Michigan to almost half (46%) four years later. African American students with at least one close friend who was not African American increased from slightly less than half (47%) at time of entrance to slightly more than half (54%) when they were seniors. ... It does not conform to the views of those in the public debate who have claimed that affirmative action has created hostile interracial ENVIRONMENTS on our college campuses."  She made the same mistake she and her surrogates are accuing me of making, in reverse.  She used "personal friendship measures" to infer a lack of hostility throughout the campus environment, when she ignored a separate question asking precisely what the environmental perception was.
So here's the predictable skinny: Everybody leaves campus thinking that race relations generally are worse, but thinking that they did a good job with their own personal race relationships. As a social scientist then, one can choose to focus on one over the other to obtain the results one wants.  Is this case really as simple as missed attribution bias (admittedly, I hadn't considered this possibility until the U-Record article)? Did the Michigan Student Study design "control" for 'attribution bias?'  Yes, in a way. It did so by including questions about both self-perception and environmental perception. They were never meant to be reported separately, as Patricia Gurin did in her expert testimony that has changed the course of history.
This is a perfect example of why the entire dataset should be available to anyone in the public or scientific community for "peer review."  Good science thrives on criticism; Gurin's science is at the moment dying on speculation.  U-M's continued insistence on withholding a dataset that it has used to influence what it claims is the biggest case since Brown v. Board reeks of a cover-up, even it it really isn't.  If U-M is truly correct that this science is well-placed, U-M has nothing to lose by releasing the entire dataset.

Here are the five final data analyses from the same file as the Michigan Student Study.

Key and timeline:
1994 Summary. The original May 24, 1994 Executive Summary or any associated files. Authors included Gerald Gurin, John Matlock, and others.
1998 Internet Summary. A 1998 version published to the Internet of the 1994 summary, but much longer, and with a more positive focus towards Michigan policy.  
Authors include Gerald Gurin, John Matlock, and Sylvia Hurtado.
1999 Patricia Gurin expert testimony.  Independent study using the Michigan Student Study (the subject of the 1994 and 1998 summaries) dataset as one of three other datasets to support testimony to the court in the admissions cases.

Photo "051"
Increased Structural Diversity (Racial Preferences) and
Increasingly Polarized Views as Students Age.

African Americans and Latinas show a slight, but statistically insignificant increased agreement with the proposition that racial enrollment diversity enhances "excellence," while Asians show a slight insignificant increased disagreement.  White students however show a statistically significant increased disagreement with enrollment diversity.  This supports the 'group identity polarization' argument I raised in my first analysis, and cuts against many of the "surveys" that U-M has published suggesting simple majorities exist supporting enrollment diversity.

This is not "surprising" data, but it runs counter to the idea, expressed by the University in different ways, that whites increasingly accept diversity programs. This result is "mixed," since other results suggest "curriculum" slightly changes white student attitudes in favor of "diversity."  This result directly contradicts notion that a "majority" of white students "increasingly support ... racial preferences."  It does not directly clash with Gurin's expert testimony to the court, but it is nonetheless important.

Photo "056"
ALL GROUPS SHOW STATISTICALLY SIGNIFICANT DECLINES IN "WHITE/BLACK" INTERRACIAL FRIENDSHIPS

All groups perceive highly statistically significant declining white/black relationships, with the White and Asian differences also being statistically significant relative to African American differences.

Arguably, one of the most damaging documents, especially to Gurin's "contact" hypothesis and her arguments in  the 1999 expert testimony.  In 1999, she never cites this data, and goes through a convoluted process to suggest a tiny improvement in black-white friendships using a variety of one year data. The closest she comes to addressing this issue with four-year data comes from a different question, that of having at least one of one's six best friends being from a race other than your own.  She testified:

"While close friendship circles of African American and white students are predominantly with peers of their own backgrounds both at entrance and after four years at the University of Michigan, there is a significant increase in the racial/ethnic diversity of such friendships."

The proportion of white students who had at least one close friend of color (among their six best friends) increased from about one third (32%) at the time they entered Michigan to almost half (46%) four years later. African American students with at least one close friend who was not African American increased from slightly less than half (47%) at time of entrance to slightly more than half (54%) when they were seniors.

"While one might hope that even more African American and white students would have increased their closest friendships with each other while at Michigan, the overall picture of interracial relationships at Michigan is predominantly positive.

It does not conform to the views of those in the public debate who have claimed that affirmative action has created hostile interracial environments on our college campuses."

First, the last paragraph seems to confuse the "personal friendship" issue with the "hostile environment" issue, at least as much as U-M accuses me of doing so.  The question on the chart below, next frame, would seem more pointed towards the question of "environmental" condition than would the "self-report" interracial friendship question.

Moreso, this analysis clearly suffers from the assymmetrical nature of the proportions of whites and blacks in the national and U-M population (she admits this).  Given that African Americans on campus are outnumbered 12 to one and the minimum requirement of having only one non-African American (not necessarily "white" either)  friend among six is asked, it is almost shockingly insignificant that African American cross-racial friendships improved from 47% to 54%.  This means that 46% of blacks have COMPLETELY "self-segregated," in an environment where 92% of their peers are of a different race, and that only 7% were influenced in some way.  Finally, considering "attribution bias" (the unconscious bias that makes it unlikely one will attribute negative behavior to oneself), it is not surprising that individuals would "self-report" their friendships to be more increasingly mixed (since U-M clearly "teaches" the notion that this is a moral good) over time but at the same time still perceive real decreasing overall campus friendship levels and increasing interracial tension.

In an example of the "cleansing" of the "updated" 1998 Internet report, the section on interracial friendships (p.25-28) completely ignores this data and the trend and spends three pages citing a variety of one year snapshots to paint a rosier picture. The 1998 Internet report comes close by suggesting that black/white relationships "were somewhat less cooperative and more personally distant," but notes, "Even here, however, it should be noted that the relationships between African Americans and White students, while distant, were not negative." (Even if "positive" using a simple "majority measure," if they get "worse" over time while at Michigan, diversity policy as a practical measure is backfiring).  This is another case of the first study using a superior analytical technique (trend reporting as opposed to one-year analysis).

Photo "057"
Increasing "Racial Conflict" on campus, all groups.
Whites and Asians show statistically significant increases, ONLY African Americans "cross-group" difference is significant.

Although all groups show increased perception of "racial conflict" on campus, and only Asians and Whites show statistically significant increases, there is a cross-group statistical signicance for African Americans relative to every other group.

Neither Gurin in 1999 nor Gurin/Matlock in 1998 address this data in a meaningful way.  The closest claim would be the assertion (that interracial hostility on campus isn't supported) described in my analysis after frame "056," but Gurin's expert testimony in 1999 cites "personal friendship data" there to refute the general thrust of this more specific data which she withheld. The fact that ONLY African American cross-group differences with other groups is statistically significant is particularly damning.  I'd argue that this data shows that race preferences are especially destructive to African American interracial relationships. 

Photo "058"
ALL GROUPS SHOW DECLINE IN RESPECT FOR STUDENTS OF COLOR BY WHITE FACULTY, Statistically Significant for White and Black Students.

All groups perceive some decline in white faculty respect for students of color, with strong (p<0.01) statistical significance amongst both whites and blacks. Although a "predominantly white institution" might be the cause of a higher than acceptable disrespect for colored students at entrance, given U-M's leadership in diversity policy and heavy emphasis on "multicultural" curriculum it is impossible to account in this way for the increased feelings of disrespect as time passes.  

This is also especially damaging to Gurin's application of the "contact" hypothesis and her arguments in  the 1999 expert testimony.  "Stigmatization" and "self-stigmatization" are occurring, and if the "contact" theory is the correct theory of pyschological change, then the opposite should be occurring.

Photo "059"
Perceptions of "University Commitment" to racial diversity polarizes over time; with White students ironically perceiving greater commitment and Blacks perceiving radically less.