Computer science is a relatively new field that traces its roots primarily to mathematics and electrical engineering. The academic traditions of computer science -- the ways a faculty member conducts research and establishes an academic reputation -- are also relatively new. Naturally, these traditions are to some degree inherited from mathematics, especially for theoretical computer scientists, and from electrical engineering, especially for computer engineers. But a substantial fraction of academic computer scientists are experimentalists. Although the natural sciences might offer a guideline for research methodologies and career development for experimentalists, the match with computer science is not good. In natural science researchers study a single given universe, whereas computation is a "synthetic field," that is, the subject matter is the creation of the human mind. Such considerations greatly influence the nature of research and the scholarly activities for experimentalist faculty. For example, the roles of theory and reproducibility are quite different in computer science compared to natural science. The question of what constitutes a solid academic reputation in experimental computer science is obviously important to the field, but the matter also becomes crucial for the academic community at large when CS faculty are hired, promoted and tenured.
The report defines "experimental computer science," explains the nature of the research questions that experimentalists study and the methodologies that they use, outlines the roles of theory and technology for an experimentalist's research, and identifies the kind of evidence -- papers, books, computational artifacts -- indicative of scholarly accomplishment in the field. The study was funded by the Computer and Information Science and Engineering (CISE) directorate of the National Science Foundation.
Computer usage in the United States is well established with a substantial fraction of the population having access to a computer connected to the Internet. But as some citizens lead almost totally electronic lives, other Americans use computers not at all or only for the most basic applications such as email. With computers costing less than televisions, and a connection to the Internet cheaper than cable, the barrier to computer usage is not so much economic as it is knowledge and expertise. So the National Science Foundatoin asked the NRC to study "What everyone should know about Information Technology" (WESKAIT) to apply IT fully in the workplace and for personally relevant purposes.
Though "computer literacy" has traditionally addressed the topic of educating non-specialists, the WESKAIT committee quickly determined that literacy is not enough. The typical computer literacy curriculum places heavy emphasis on skill-level instruction to achieve proficiency with computer applications, such as email and word processing. But this "click here, click there" type of education does not have the staying power to serve students well through rapid technological changes that are characteristic of IT. The skills soon become outdated as new applications and even new versions of current applications are introduced. Because people cannot continually take literacy courses to upgrade their knowledge, it is imperative that they become lifelong learners of IT, that is, to keep up by teaching themselves. The knowledge level needed to be a lifelong learner of IT was dubbed "fluency."
The report identified three types of knowledge that could enable a person to be Fluent: Skills, Concepts and Capabilities. The three types of knowledge are independent and orthogonal. The Skills embody proficiency with contemporary applications, e.g. email, word processing, etc., and they correspond to traditional computer literacy content. Thus, Fluency contains literacy. Concepts refer to an understanding of the intellectual underpinnings of IT drawn from its constituent fields, e.g. how a network transmits information, how information is structured, etc. Concepts provide the foundation for future learning. Capabilities are higher-level thinking in the context of IT, e.g. problem solving, logical reasoning, etc. Capabilities enable users to be successful at applying IT. The report listed the top ten most important topics in the three classifications.
From 1995 through 2001 I served on ARLTAB, the Army Research Lab's Technical Assessment Board. The purpose of ARLTAB is to understand and evaluate the technical quality of ARL scientists' research. Only independent peer review can assure high quality research, but the usual mechanisms of science don't all apply at ARL. For example, compared with other labs charged with conducting fundamental research, ARL scientists publish relatively fewer papers, not because the work is classified, which it only occassionally is, but because of the nature of the topics, which may not be of wide interest in their technical communities. The ARLTAB provides the "quality assessment" for ARL.
ARLTAB is composed of six panels. In addition to being on the board, I contributed to the Digitization Panel, which I chaired for a time. The ARLTAB has issues three book length reports assessing ARL's scientific quality. These reports are available by contacting the ARLTAB staff at the NRC.