Safety and the Human-Computer Interface

Nancy Leveson

In avionics and other high-tech systems, computers are no longer simply reading sensors, integrating the information, and displaying it for operators to use. Many systems today provide either shared control between the computer and the operator, or total computer control with the operator controlling or monitoring the computer (rather than the process itself). Recent accidents in commercial aircraft and other complex systems have resulted from difficulties in integrating computer and human control.

Mode confusion is a good example of the problems. Mode confusion occurs in systems that allow the same operation to have different effects depending on the current system mode. Mode errors occur in simple systems when an operator acts in a way appropriate to one mode when the system actually is in another mode. Advanced automation systems, where computers implement control actions of their own, add new types of mode-related problems because the system status and mode (and thus behavior) can change independent of direct and immediate operator commands. Controllers in these systems, such as pilots, have the added cognitive task of keeping track of the computer-directed mode changes. Problems are also arising because the operators do not understand the logic of the automated systems.

The safe design of the human-computer interface and cockpit procedures is dependent on appropriate design of the software, and appropriate design of the software can only be assured in the context of the operator tasks and cognitive abilities. We are developing techniques and tools that assist in design and verification of the coordination, interaction, and interfaces between system components -- human, hardware, and software.

Our current research goal is to identify design constraints on the automation based on known cognitive constraints on the human operator and engineered or natural environmental constraints. The first step in accomplishing this goal is to identify the types of errors that humans make in highly automated systems. Using this information, we can analyze the blackbox behavior specified in the automation requirements to predict where errors will occur and use this information to design the automation and the operator procedures, tasks, and interface. At first, we are simply going to analyze current designs, but our long term goal is to identify software design criteria and techniques that will help to create better designs from the beginning.

RECENT PAPERS

Analyzing Software Specifications for Mode Confusion Potential, by Nancy G. Leveson, L. Denise Pinnel, Sean David Sandys, Shuichi Koga, Jon Damon Reese. Presented at the Workshop on Human Error and System Development, Glascow, March 1997. (Postscript).

Increased automation in complex systems has led to changes in the human controller's role and to new types of technology-induced human error. Attempts to mitigate these errors have primarily involved giving more authority to the automation, enhancing operator training, or changing the interface. While these responses may be reasonable under many circumstances, an alternative is to redesign the automation in ways that do not reduce necessary or desirable functionality or to change functionality where the tradeoffs are judged to be acceptable. This paper describes an approach to detecting error-prone automation features early in the development process while significant changes can still be made to the conceptual design of the system. The software requirements are modeled using a hierarchical state machine language and then analyzed (manually or with automated assistance) to identify violations of a set of design constraints associated with mode-confusion errors. The approach is illustrated with a model of the software controlling a NASA robot.

Designing Automation to Reduce Operator Errors by Nancy G. Leveson and Everett Palmer (NASA Ames Research Center). In the Proceedings of Systems, Man, and Cybernetics Conference, Oct. 1997 (PostScript)

Advanced automation has been accompanied, particularly in aircraft, with a proliferation of modes, where modes define mutually exclusive sets of system behavior. The new mode-rich systems provide flexibility and enhanced capabilities, but they also increase the need for and difficulty of maintaining mode awareness. A previous paper described some categories of potential design flaws that can lead to mode confusion errors and described an approach to finding these flaws by first modeling blackbox software behavior and then using analysis methods and tools to assist in searching the models for predictable error forms, i.e., for automation features that can contribute to operator mistakes. This paper shows an example of the approach for one particular feature, i.e., indirect mode changes, using an example from the MD-88 control logic. The particular indirect mode transition problem used in the example, called a ``kill-the-capture bust'' has been noted in many ASRS incident reports.