Keunwoo Lee (klee@cs)
This article provides a brief introduction to ethnography and its applications in software engineering requirements. A small collection of references to literature is also presented.
Programming is labor-intensive and--for a variety of reasons--most serious project delays manifest themselves by code not being ready to ship. So, why not eliminate programming as an activity altogether?
To many managers, getitng rid of the arrogant, underdisciplined, over-paid, technology-obsessed, improperly dressed, etc. programmers would appear to be a significant added benefit.
- Bjarne Stroustrup [C++97]
Stroustrup goes on to point out that--alas!--the manager's holy grail eludes us yet1. Naturally, the dual of the above suggestion would be to eliminate the managers (a reform often suggested by programmers, and not in jest).
And, while we're eliminating inconvenient factors from the equation, perhaps it might be a good idea to eliminate the client as well. After all, clients are ignorant of software engineering and hence make unreasonable demands; they do not know precisely what they require, and hence are never satisfied, even with systems built to specification; and they have no clue how to communicate their requirements effectively, so even when the client knows what (s)he wants, the systems engineer must go through all manner of contortions to extract those requirements in some useful form.
Life would be nice without clients; but on the other hand software engineers would be broke. Similar caveats hold for the proposals to eliminate programmers, managers, or other human factors from software systems development. Therefore, software engineering must concern itself with human systems as much as with software systems.
Of course, disciplines devoted solely to the study of human systems exist: anthropology, psychology, ``cultural studies'', etc. It seems obvious that we should deploy the methods developed within these disciplines in the service of software engineering.
In this case, there are two questions one must ask. First of all, which methods and approaches from social science can be adopted by software engineers? Second, which phases or subfields of software engineering could benefit from the application of these methods?
The combinations of answers to these questions delineate a large space of possibilities. Methodologically, the social sciences encompass a broad collection of disciplines and competing theories; the spectrum of possible applications in software engineering is similarly broad. For example, one can imagine applying psychology to the study of developer preferences for particular design philosophies, or the use of sociology to examine statistical patterns across software development teams of different demographic compositions. The number of imaginable combinations of social and computer sciences is nearly unlimited. In researching this article, I even came across one paper that discussed the use of ethnographic observations to suggest new approaches (perhaps leading eventually to algorithms) for indexing and categorizing text documents [Case88].
Therefore, even a brief survey of all work to date that bridges social science and software engineering would fill a book. Therefore, this article will focus on one particular node of connection between social sciences and software engineering: the methodology of ethnography, and its application to requirements engineering.
I first present a general introduction to ethnography, including a brief discussion of methodological and theoretical issues. Next, I will present a case study from the literature. Finally, I will present a survey (non-comprehensive, obviously) of sources that should provide accessible starting points for the interested researcher or student.
Ethnography is a specific collection of practices for observing human behavior in social settings. Blomberg [Blomberg95] provides a cogent definition of ethnography for computer scientists:
As employed in the HCI and system design communities, ethnography most often refers to an approach used to develop understandings of everyday work practices and technologies in use. While there is a great deal of variability subsumed in the practice of ethnography, most practitioners share a few basic presuppositions. These include: a commitment to studying activities in the particular settings in which they occur; an interest in developing detailed descriptions of the lived experience; a focus on what people actually do, not simply on their accounts of behaviour; and a concern with understanding the relation of particular activities to the constellation of activities that characterize a setting.We can therefore distinguish ethnography from other social research methods by its emphasis on field observation. An ethnographer does not bring the world into the laboratory, but rather makes the world into the laboratory. Ethnography originated among anthropologists who wished to study cultures vastly different from that of the anthropologist--for example, island aboriginal cultures. The people and environments native to these cultures were only accessible to researchers who physically placed themselves in the field for an extended time (often years), in order to gather a large corpus of firsthand observations about a foreign culture in its ``natural'' context.
Anthropology no longer restricts its domain to the study of distant or ``exotic'' cultures, of course. Ethnography's lens has been turned inward, on subcultures within Western societies; and as Blomberg's quote implies, ethnography has already percolated down to the analysis of human-computer interaction and systems design.
Ethnographers employ a number of observational methods to collect data. One example technique, which may convey the flavor of the data generated used by ethnographic methods, is the creation of textual transcripts of conversations. Such transcripts begin with the ethnographer setting up some mechanical recording equipment (audio and possibly video) at the site of activity. Afterwards, the ethnographer will painstakingly go over the recorded data and transcribe it to a textual form that contains data not only on the words spoken, but specific textual cues indicating pauses, changes in vocal tone, etc. Chapter 5 of [Suchman87] provides several examples of the end product of this technique; here is one excerpt:
C: .hhhh aa::of course under the circumstances Dee I would never:: again permit im tuh see im.In this notation, colons (:), for example, represent pauses in midphrase, with more colons representing a longer pause. The left bracket ([) indicates that the speaker began speaking before the previous one was done. The parenthesized 0.7 indicates the transcriber's level of uncertainty.
D: Yeah.
(0.7)
C: tlk. Be:cuz he--
[
D: Wul did'e ever git--ma:rried'r anything?
C: Hu::h?
Not all ethnographic data resembles the above, but the precision, level of detail, and emphasis on how the participants speak as much as what they speak, should be obvious. The above transcript differs greatly the end product differs from the account that a layperson would give of the conversation--or, more to the point, that a software engineer would give. Readers interested in further discussion of systematic observational methods for social research may wish to consult [Weick85], which provides an overview (of bogglingly wide-ranging scope).
In the introduction, I mentioned, somewhat facetiously, the impossibility of removing human factors from software engineering. However, this broad fact, which few would deny, does not in itself constitute an argument that ethnography in particular is useful for requirements engineering.
What, then, are the specific merits of ethnography for addressing requirements? How might ethnography mitigate or solve some of the client relationship problems raised at the beginning of this article?
Ethnography for software engineering (which, for convenience, I will refer to as ESE in the remainder of this article), dates at least to the early 1980's. At that time, a number of researchers began to realize that software was always used in the context of social situations, and hence it would be worthwhile, when evaluating software, to study users in social contexts. [Suchman87], based on early research done at Xerox PARC, remains a seminal work in the field; Suchman focuses on HCI rather than requirements, but the two are related so it may be illuminating to describe Suchman's work here.
Suchman puts forth the notion that human actions are always ``situated''--that is, human ``plans'' for action are always vague and provisional, and that dialogue with both other humans and machines always involves some degree of situational adaptation and repair for misunderstandings. Therefore, Suchman asserts, machine systems that interact with humans must accommodate such repair and adaptation in a way that matches the implicit structures of human conversation. In the second half of the book, Suchman presents a case study of how one photocopier system fails in this respect, applying the ethnographic methods presented in the first half to observe two users' interaction with the copier.
Over the course of the photocopier study, it becomes clear that users expect certain ``conversational conventions'' that system fails to satisfy. For example, Suchman describes one interaction sequence in which the copying machine requests (via a screen display) that a user perform exactly the same action twice--in other words, a standard loop, which seems trivially obvious to any programmer. Suchman then points out that in human conversation, there is no such action; a repetition of exactly the same sentence twice consecutively (for example, saying ``hello'' multiple times when answering a telephone) is usually not interpreted as normal speech but as an indication that an error has occurred in communication. The prototypical response to such an error is to say, ``Can you hear me?'' The users who were confronted with the copying machine's identical consecutive messages were therefore confused as to whether the machine had understood their previous action.
This discrepancy between machine and human expectations would not have been obvious without ethnographic methods and their supporting theoretical foundation in studies of human-to-human interaction. Ethnography, in other words, provides deeper insight into human behavioral patterns, insights that do not surface by pure deduction or even during ``end user testing''2 in the lab.
As software, and hence human-computer interactions, become more complex, human expectations of the software product's sophistication will increase accordingly. It clearly becomes more important for designers to be conscious of what humans expect from a software product.
The ethnographic research performed by Suchman indicates that ethnographic methods can yield insights into the success or failure of existing designs. But why wait until a system has been deployed to evaluate its effects on people? Applying ethnography in the earliest stages of software development could yield fundamental human factors insights before difficult, opaque, or even useless systems are designed and implemented in the first place.
If this is the case, then we have entered the realm of requirements. The case for integrating ethnography into requirements is therefore simple: if studying the users of a system makes the system demonstrably more usable, then examining the practices and needs of system users during the requirements phase can eliminate costly misinvestment later on. And if ethnography provides software engineers with novel and comprehensive insights that other techniques omit, then clearly there would be some benefits in applying it.
These arguments may still seem rather abstract; hopefully, presentation of a case study will make the argument more persuasive. However, before moving on to the case study, we should consider the disadvantages of ethnography.
No software engineering tool is without caveats, and of course ethnography has its own particular set.
For example, I mentioned previously that originally, ethnographers would spend years in the field studying a culture before drawing conclusions. Such time scales, of course, are completely impractical for most projects in an industry where major technologies must be developed and go to market in a matter of months. Furthermore, even when ethnographic study does not span years, rigorous examination of certain practices is extremely labor-intensive. Transcribing an hour-long conversation or interaction, for example, can take many hours of labor by the ethnographer.
In short, ethnography requires good data, and good data are expensive. How, then, can the average software development organization possibly afford to practice ethnography for requirements?
There are two replies to this question. First, every software engineering paper I have ever read has convinced me of the need for greater basic research into the culture and practices of software development and client interactions. Basic research, funded by universities or large research labs like Xerox PARC (where Suchman did her work), can afford to focus on long-term research projects whose scope, and hence interest, outlasts particular software projects or even particular companies.
Second, in the context of applied research, ethnographic tools can still be employed without the use of the full ethnographic techniques. [VilSom99] advocates a two-tiered methodology wherein initial ethnographic surveys yield rapidly collected but incomplete results. In-depth surveys that require more labor can then be performed, with domains limited to the problem areas that were isolated in the first pass.
Clearly, this ``zooming'' approach (as Viller and Sommerville call it) will yield inferior data compared to a uniformly comprehensive approach. Problem areas may be missed in the initial survey and hence ignored by later passes; the limited scope of the later passes may distort the researchers' view of the findings; the compressed time-scale may inflate the effect of statistical noise in the data. However, it seems clear that some examination of users is preferable to none.
Incidentally, apart from concerns of cost, one might also raise the objection that that talented ethnographers with an understanding of software systems are even rarer than skilled software engineers. In other words, there is a clear supply shortage, entirely apart from the demand question. In the short term, unless demand for ``indsustrial ethnography'' increases and draws in a large pool of talented people, small organizations might have only a limited opportunity to attract and employ ethnographers.
No matter how compelling ethnography seems in the abstract, we may not be convinced of its effectiveness without concrete evidence of its results when applied. In this section, I present a case study from the literature that, hopefully, both conveys the flavor of applied ethnography and presents convincing evidence that it can improve the requirements engineering process.
The Work-Oriented Design study in [Blomberg95] attempts to ``explore the possibility of bringing site-specific studies of work into a productive relation with technology design''. In other words, there has been some trepidation in the ``pure'' ethnographic community towards prescriptive (normative), as opposed to descriptive (positive) research. Blomberg et. al. explore the possibility of applying ethnographic methods directly in the system engineering process. Ultimately, these methods cut across multiple phases of software engineering as traditionally viewed, but examining solely the effect on requirements elicitation is quite revealing.
The software system in question was intended for a Silicon Valley law firm with over 500 employees (roughly 40% attorneys and 60% support). The firm expressed interest in developing an application to assist them with document management, due to the extremely large volume of paper documents used, and agreed to participate in the study. Notably, no firm guarantee was ever given that the project would deliver a working product for use--the firm consented because, Blomberg writes, they ``simply were offered an opportunity to reflect with us on their current work paractices, to preview innovative technology directions, and to help shape future products''. One suspects the firm hoped to get a system out of the deal as well, of course, but the institutional receptiveness to the ethnographic researchers may well have played a factor in the researchers' success.
The document processing needs of the firm can be summarized as follows:
You have, you know, 300 cartons of documents and you tear through them and say, I'm going to put Post-Its on the ones we have to turn over. And then, ideally, you hire chimpanzees to type in From, To, Date. And then, ideally, you then have lawyers go through it again and read each document, with their brains turned on.The attorneys therefore suggested that a nearly completely automated document coding system could solve the problem.
At first blush, and even after some consideration, a software engineer might not find much fault with this assertion. If the documents are indeed all reasonably marked, perhaps a scanning and OCR application could automate the data entry portion of the process, with document coders relegated to correcting occasional errors.
Deeper interviews, however, revealed that the attorneys' viewpoint was deeply flawed. Interviews with the supervisor of support directly contradicted this simplistic view of the ``document coding'' process:
[The supervisor] mentioned that coding even such mundane information as the date of the document was not always straightforward. A document might have multiple dates; the date it was written, the date it was signed, and the date it was faxed. Deciding which date to code required knowledge of the case and interpretation of a document's content and relevance ...[Furthermore] Coders are simply presented with boxes of ``paper''. The paper clips that sometimes joined pages together cannot be assumed to hold a single document. Deciding to code the pages joined together as single documents, multiple documents, or a single document with multiple attachments might require reading the text.Therefore, early in the requirements phase the fantasy of automating the coding was discarded.
Of course, at this point, ethnography still had not contributed anything in particular that might not have been discovered in other requirements methodologies. However, other requirements elicitation techniques might have found the flaw later, at a point when significant cost had been invested in the attorneys' original vision. After all, the directors of the firm were probably attorneys, and therefore the main point of contact between (for example) an IT consulting firm's managers and the law firm. The software developers, receiving orders from their managers, might therefore have naively given credence to the attorneys' characterization of the work and simply gone along with the plan until reality set in.
Furthermore, other requirements processes would not have suggested a direction for revising the system requirements once the original misapprehension was discovered. Ethnographic methods suggest a definite framework for progress towards a better requirement: examine the people who do the work, and determine their problems.
At the time, the documents coding supervisor was experimenting with new systems for entering the coded forms into a database. At first the coders marked document metadata with attached forms, which data entry personnel then used to enter data into a database; then there was an experiment to use highlighter pens and various other devices to mark photocopies of the documents directly; then, when problems in this process arose, the forms procedure was brought back.
The ethnographers observed the various forms these experiments took, and used their observations directly to inform a new set of requirements. The WOD developers then produced a prototype that addressed the specific working conditions of the coders (for example, providing multiple methods for entering documents into the system, and multiple methods of annotating a document). The paper does not discuss whether this prototype was ever augmented into a fully working system.
In any case, Blomberg summarizes the requirements driving the design philosophy as follows:
Because of our recognition that a great deal of judgment is exercised by document coders, the design did not attempt to automate the coding process. Instead, the idea was to minimize the need to directly transcribe (copy by hand) information from the document to the paper form and to maximize the ease with which the database could be checked for accuracy.In other words, the result is nearly diametrically opposed to the attorneys' initial characterization of an ideal system. Rather than providing a rigid automation to replace the coders, it provides a flexible system to support the coders.
Therefore, we see that integrating ethnography into the software engineering process from the very beginning resulted in a radical change in the end product. It resulted in the discarding of unproductive approaches early in the process, when doing so is cheap, and it provided a clear direction for formulating requirements in the face of conflicting information.
From a software engineer's perspective, one unfortunate inconvenience of researching the work of social scientists is that they are much less likely to be conversant in digital document publishing. Therefore, the reader is advised that tracking down work by leading ethnographers will often require the elicitation of dead trees from physical, corporeal libraries.
The reader may have noticed that I quote [Blomberg95] extensively in this article. Blomberg's paper on the Work-Oriented Design project serves well as an excellent shorter introduction to ESE. This paper also has the benefit of a decade of hindsight on the topic, and hence contains a historical summary of developments in ESE research and literature.
Lucille Suchman, one of Blomberg's collaborators on the Work-Oriented Design paper, was one of the earliest researchers to explore the HCI/ethnography boundary. Her work from 1980's onward is invariably described as ``seminal''3; [Suchman87] serves as a good introduction to theoretical insights into HCI stemming from ethnographic study.
Until recently, ESE has (as noted earlier) generally focused on HCI and hence the design and user-interface aspect of software engineering (though much of what the ethnographers call ``design'' would be called ``requirements'' by a software engineer today). The work by Suchman and Blomberg certainly focuses on those areas, probably because HCI is the area where ESE can most obviously make contributions. In recent years, however, there has been a realization of the broader uses of ESE. The Coherence project focuses on requirements engineering rather than design, though their published materials are less extensive than older ESE work. The Coherence home page, which has links to the project's papers, can be found at:
Additional resources used in my background research can be found in the references section.
A note on the IEEE papers: the IEEE, unlike the ACM, encodes session-specific information in the URL to their secure server. Therefore, I am unable to provide direct links to papers from IEEE publications. Where available, links to abstracts on the IEEE non-subscriber server are provided.
Ethnography was first proposed as a technique for improving software engineering processes in the early 80's. Research results encouraged others to follow, and although the method has been gaining supporters it appears to be used principally in software engineering research projects. The scarcity of corporate headhunting for ethnographers suggests that the techniques have not reached mainstream acceptance.
In any case, ESE clearly does not deserve to remain purely a niche technique, for researchers only. For all the reasons given in this article, it is clear that software engineering in general, and requirements engineering in particular, has much to gain from cross-breeding with ethnography.
Thanks to William Griswold (UCSD/Xerox PARC) and Janelle Taylor (UW Dept. of Anthropology) for their invaluable help in pointing me to references.
This document was generated using the LaTeX2HTML translator Version 99.2beta6 (1.42)
Copyright © 1993, 1994, 1995, 1996,
Nikos Drakos,
Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999,
Ross Moore,
Mathematics Department, Macquarie University, Sydney.
The command line arguments were:
latex2html -nonav -split 0 ethno.tex
The translation was initiated by Keunwoo Lee on 2000-07-01