Erroneous NRC Ranking Data for UW CSE
The NRC assessment of UW Computer Science & Engineering is based on clearly erroneous data. The assessment is meaningless, and in no way representative of the accomplishments of UW CSE.
Errors in the data affect (at least) UW CSE, many other computer science
programs nationally, and many programs in other
fields at the University of Washington.
During the week of September 19th, NRC provided pre-release access to its long-delayed "Data-Based Assessment of Research-Doctorate Programs in the United States," scheduled for public release during the week of September 26th.
We, along with colleagues in other computer science programs nationally and colleagues in programs in other fields at the University of Washington, quickly discovered significant flaws of three types in NRC's data:
- Instances in which the data reported by NRC is demonstrably incorrect, sometimes by very substantial margins.
- Instances in which the accuracy of the data cannot easily be checked, but it does not pass even a rudimentary sanity check.
- Instances in which institutions interpreted NRC's data reporting guidelines differently, yielding major inconsistencies.
Here are three specific examples affecting UW CSE:
Due to difficulty in interpreting NRC's instructions, NRC was provided with an incorrect faculty list for our program - essentially, a list that included anyone who had served as a member of a Ph.D. committee. In 2006 (the reporting year), UW CSE had roughly 40 faculty members by any reasonable definition. In the NRC study, our "total faculty" size is listed as 91 and our "allocated faculty size" (roughly, full time equivalent) as 62.5. A large number of these "additional faculty" were industrial colleagues - whose "academic records" (including grants, publications, and awards) were quantitatively evaluated by NRC as if these individuals were full members of our faculty. Since faculty size is the denominator in many measures computed by NRC, you can imagine the result - clearly erroneous. (A correct list of UW CSE "core
faculty" as of 2006 is
- NRC's pre-release data reported UW CSE with 0% of graduate students "having academic plans" for 2001-05 (the reporting period for this measure). In fact, roughly 40% of our graduating Ph.D. students took full-time faculty positions during this period. We are one of the top programs nationally in producing faculty members for major departments; in recent years our graduates have taken faculty positions at Berkeley, CMU, MIT, Princeton, Cornell, Wisconsin, Illinois, Michigan, Penn, Waterloo, Toronto, WashU, UCSD, Northwestern, UCLA, UBC, Maryland, Georgia Tech, UMass-Amherst, and many other outstanding programs. NRC obtained this number from an outside data provider; it's clearly erroneous.
[Update: A calculation error was fixed prior to the public release
of the data on September 28th. The reported value for UW CSE
is now 25% - still erroneous, but heading in the right direction.
We understand that the remaining error is due to the use of a
notoriously unreliable data source for this quantity, rather
than the institutional data that NRC collected.]
- NRC reports UW CSE as having 0.09 "awards per allocated faculty member." The erroneous faculty count is not sufficient to explain this, given that our faculty includes
fifteen Sloan Research Fellowship recipients,
eight Fulbright recipients,
six PFF/PECASE recipients,
two NAE members,
two Guggenheim recipients,
one MacArthur Fellow,
one Fellow of the American Academy of Arts & Sciences,
thirty one Fellows of major professional societies,
and twenty seven NSF CAREER Award winners.
Our own comparison of 2006 faculty awards to the NRC
award list indicates that the value reported for our
department is low by roughly a factor of ten! This is
only an approximation, but it indicates that the NRC
data is clearly erroneous.
(See additional information on this topic
The University of Washington reported these issues to NRC when the pre-release data was made available, and asked NRC to make corrections prior to public release. NRC declined to do so (with the exception of the one bug-fix noted above). We and others detected and reported many other anomalies and inaccuracies in the data during the pre-release week.
The widespread availability of the badly flawed pre-release data within the academic community, and NRC's resolve to move forward with the public release of this badly flawed data, have caused us and others to urge caution - hence this statement.
For our program - and surely for many others - the NRC assessment is based on data that is clearly erroneous.
The results are meaningless.
Statement from the Computing Research Association: "CRA
has serious concerns about the accuracy and
consistency of the data being used in the evaluation of the
Computer Science discipline."
Statement from the University of Washington College of Engineering: "We
we are greatly concerned by clear inaccuracies in the data used
in the report."
Statement on "Faculty Awards" from UW Computer
Science & Engineering: "Moving beyond the significant flaws
in award selection and classification, one quickly arrives at the
sorts of data inaccuracies that plague other aspects of the NRC assessment."
Computer Science & Engineering|
University of Washington
Seattle, WA 98195-2350
(206) 543-1695 voice, (206) 543-2969 FAX
[comments to Hank Levy]