Joint Cognitive Systems Research Articles

Design of joint systems: a theoretical challenge for cognitive systems engineering

In this paper the “joint systems approach” will be discussed. The approach is considered as a new paradigm in cognitive systems engineering (CSE). Its central idea, that human and technology form a functional unity, is as such not new. Why, then, has this idea become so appealing right now? In the first part of the paper we seek answers to this question by analysing the current situation in product design. The conclusion is that the focus of design is shifting from single products to intelligent environments. This change in focus induces tensions into the design process that urge for solutions. One of them is re-conceptualising the relationship between human and technology, precisely what the joint system approach is about. Three different joint system approaches are considered: the joint cognitive systems approach (JCS), the Riso extended CSE approach, and the joint intelligent systems (JIS) approach proposed by us. Comparisons are made with regard to how these approaches understand the joint system to be organised. While the JCS approach focuses on the human–technology relationship, the other two consider this relationship as embedded in a context of object-oriented activity. As a consequence, environment becomes included in the joint system. In JIS approach we propose the use of the semiotic concept of habit to characterise the ways of functioning of the joint system. “Habit” is a tool for identifying generic patterns in the situation-specific behaviour of the system. Defining habits enables expressing the meaning or purpose of the system’s functioning. In the end of the paper, we propose a design process model for the development of JIS. This approach is aimed at designing systems in usage i.e. to design of practices.

Expanding Functional Analysis to Develop Requirements for the Design of the HumanComputer Interface

AbstractThe purpose of this paper is to provide a framework for extending the functional analysis of a system to the human decision‐making agents in order to improve overall system effectiveness. The subsystems needed to accomplish the goals of the system can often be found through a functional analysis. However, the different subsystems need to be completely integrated in order for the system to be effective. This integration is often problematic within the Human‐Computer Interface (HCI), where the operator is reliant on the information provided in order to have an understanding of what is occurring within the work domain. As a functional analysis helps to define the goals to be accomplished and the subsystems that accomplish them, it follows that expanding the depth of the functional analysis will help integrate the human operator and technology through the HCI to make a more effective Joint Cognitive System (JCS). The functional analysis leads to the creation of more insightful design requirements for the HCI that directly link the work that the user must accomplish and the information needed to complete this work – making it truly user‐centered design. An example of how the functional analysis can be applied to a system is provided, comparing an HCI developed with typical design requirements to an HCI developed from an expanded functional analysis.

Flightdeck and Air Traffic Control Collaboration Evaluation (FACE): Evaluating aviation communication in the laboratory and field

The challenge to anticipate the human factors impact of introducing new technologies into a safety critical environment can be addressed in a number of ways. This paper presents a research programme that utilised both laboratory- and field-based assessments to examine the way in which datalink and freeflight may affect the communication and collaboration between pilots, air traffic controllers, and other actors and artefacts in the flightdeck–air traffic control (ATC) joint cognitive system. An overview of the results from these studies is presented, and guidance is provided as to the likely situations in which this new technology is most likely to be successfully applied. In addition, the methodological approach of combining results from field and laboratory data is discussed.

Flight decks and free flight: Where are the system boundaries?

The change from managed to free flight is expected to have large effects, over and above the intended efficiency gains. Human factor concerns have understandably focused on how free flight may affect the pilots in the cockpit. Yet it is necessary to see the change from managed to free flight as more than just an increment to the pilots’ work. Despite the best intentions the transition will not be a case of a smooth, carefully planned and therefore uneventful introduction of a new technology. It is more likely to be a substantial change to an already challenging working environment, in the air as well as on the ground. The significant effects will therefore not just happen within the existing structure or distribution of work and responsibilities, but affect the structure of work itself. This paper takes a look at free flight from a cognitive systems engineering perspective and identifies two major concerns: first what effects free flight has on the boundaries of the joint cognitive systems, and second how this affects demands to control. The conclusion is that both will change considerably and that we need to understand the nature of these changes before focusing on the possible effects of free flight on pilots’ performance.

Evaluating the Effectiveness of a Joint Cognitive System: Metrics, Techniques, and Frameworks

An implication of Cognitive Systems Engineering is that joint cognitive systems (JCS; also known as complex socio-technical systems) need to be evaluated for its effectiveness in performing the complex cognitive work requirements. This requires using measures that go well beyond “typical” performance metrics such as the number of subtask goals achieved per person per unit of time and the corresponding simple baseline comparisons or workload assessment metrics. This JCS perspective implies that the system must be designed and evaluated from the perspective of the shift in role of the human supervisor. This imposes new types of requirements on the human operator. Previous research in CSE and our own experience has lead us to identify a set of generic JCS support requirements that apply to cognitive work by any cognitive agent or any set of cognitive agents, including teams of people and machine agents. Metrics will have to reflect such phenomena as “teamwork” or “resilience” of a JCS. This places new burdens on evaluation techniques and frameworks, since metrics should be generated from a principled approach and based on fundamental principles of interest to the designers of the JCS. An implication of the JCS perspective is that complex and cognitive systems need to be evaluated for usability, usefulness, and understandability; each of which goes well beyond raw performance. However, conceptually-grounded evaluation frameworks, corresponding operational techniques, and corresponding measures for these are limited. Therefore, in order to advance the state of the field, we have gathered a set of researchers and practitioners to present recent evaluation work to stimulate discussion.

1.1.4 Joint Cognitive Systems: Considering the User and Technology as One System

AbstractWith the addition of advanced technologies and automation within systems, the tasks that operators are asked to complete have grown more complex. This has led to greater opportunities for malfunctions, critical incidents, and catastrophes. A major contributor to these problems is the human operator and the technology not being explicitly designed as a single Joint Cognitive System (JCS). The operator is typically only blithely considered during system design and is typically expected to adapt to inputs from the other subsystems. Thinking of the operator and the automation/technology from a holistic perspective, as a single JCS, instead of separate subsystems, allows for more effective, agile, and robust decision‐making. The operator must have a complete understanding of the automated processes and the work domain, leading to more informed decision‐making. Thinking of the entities as co‐agents, rather than two isolated components that must interface, allows for more effective problem solving and decision‐making.

Decision-Centered Testing (DCT): Evaluating Joint Human-Computer Cognitive Work

In order to test the effectiveness of a human operator paired with a decision support system, it is necessary to complement current testing practices addressing software validation, human performance, and usability. Decision Centered Testing (DCT) aims at testing the effectiveness of operators teamed with Decision Support Systems (DSS) in any challenging work domain. DCT is grounded in a Cognitive Systems Engineering (CSE) framework, where the concept of a joint cognitive system (JCS) is central. DCT aims at evaluating the decision-making effectiveness across identified 'error prone' regions in the JCS structure. A description of the DCT Methodology with an illustration taken from an initial application of the methodology is presented. In this application, insights from the DCT methodology enabled the definition of appropriate test metrics and the construction of unique test scenarios to exercise the decision-making effectiveness. From this application, it can be concluded that following the DCT Methodology facilitated the construction of an evaluation framework for assessing JCS net decision-making effectiveness.

The Role of Automation in Joint Cognitive Systems

Automation is extensively used in present-day technological systems, but often with little concern for the people who must use it. The classical automation principles, the left-over principle and the comparison principle, both assume that human functions can be replaced by technology with few adverse effects. This has caused a gradual removal of the operator from the process, with a concomitant loss of the ability to control. A more contemporary view, based on the principles of cognitive systems engineering, considers automation from the perspective of the joint system and uses automation to enhance the system's ability to maintain control of the process.

Human Factors in the Design of Supervisory Control Systems and Human-Machine Interfaces for Highly Automated Complex Systems

The design of control systems and human–machine interfaces in the field of complex and safety-critical environments remains today an open issue, in spite of the high technological evolution of the last decades. The increasing use of automation has improved efficiency, safety and ease of operations but, at the same time, it has complicated operators’ situation awareness and has changed the nature of their possible errors. The research activity described in this paper is an attempt to develop a methodological framework to support designers of control systems and human–machine interfaces. In particular, it focuses on the need for a deeply recursive approach related to the implementation of the systemic and human aspects of the design process of a human–machine system, intended as a Joint Cognitive System. A validating case study has been performed, based on the full application of the framework on the control of the turbine/alternator system of a thermoelectric power plant in northern Italy.

Factors influencing analysis of complex cognitive tasks: a framework and example from industrial process control.

We propose that considering four categories of task factors can facilitate knowledge elicitation efforts in the analysis of complex cognitive tasks: materials, strategies, knowledge characteristics, and goals. A study was conducted to examine the effects of altering aspects of two of these task categories on problem-solving behavior across skill levels: materials and goals. Two versions of an applied engineering problem were presented to expert, intermediate, and novice participants. Participants were to minimize the cost of running a steam generation facility by adjusting steam generation levels and flows. One version was cast in the form of a dynamic, computer-based simulation that provided immediate feedback on flows, costs, and constraint violations, thus incorporating key variable dynamics of the problem context. The other version was cast as a static computer-based model, with no dynamic components, cost feedback, or constraint checking. Experts performed better than the other groups across material conditions, and, when required, the presentation of the goal assisted the experts more than the other groups. The static group generated richer protocols than the dynamic group, but the dynamic group solved the problem in significantly less time. Little effect of feedback was found for intermediates, and none for novices. We conclude that demonstrating differences in performance in this task requires different materials than explicating underlying knowledge that leads to performance. We also conclude that substantial knowledge is required to exploit the information yielded by the dynamic form of the task or the explicit solution goal. This simple model can help to identify the contextual factors that influence elicitation and specification of knowledge, which is essential in the engineering of joint cognitive systems.

Organisational Appropriation of Technology: A Case Study

This paper investigates how organisations – as joint cognitive systems – deal with change. This is explored in a case study into the use of web-based technology in an organisation and the ways in which its current and future use may be influenced by the local concerns of the actors with it. The paper looks at some of the factors that need to be taken into account for improving technology, and more specifically, in intranet development. In particular, the paper discusses the case experience in the light of previous research in IT-enabled organisational transformation and contributes to our understanding of human interaction with network technologies in an organisational context.

Designing for operations: Towards a sociotechnical systems and cognitive engineering approach to concurrent engineering

In the practice of concurrent engineering, the factors that are considered early in the product design process include manufacturability, assembly, and cost. A set of issues that are not typically considered revolve around the operational requirements for human workers in the manufacturing system. What tasks will human workers accomplish? How will these tasks be organized and coordinated? What information and resources need to be shared? Will the workers have a coherent set of job responsibilities? How should the manufacturing environment be designed to support effective work practices? How can a manufacturing process be designed that also informs organizational structure and takes into account the quality of working life? The field of sociotechnical systems theory (STS) focuses on exactly these kinds of issues. Rather than subscribing to the usual view of technological determinism — that a complex human-machine system is designed solely with respect to optimization of technical criteria — the goal of STS is to jointly optimize both human and technological considerations in system design and operation. The spirit of STS has much in common with recent work in cognitive systems engineering that advocates the design of joint cognitive systems in which machines serve as flexible, context-sensitive resources for human problem solving. Furthermore, a focus on design teams necessitates the study of the relationship between group work and technology as studied in the field of computer-supported cooperative work (CSCW). This paper briefly reviews current research in sociotechnical systems theory, computer-supported cooperative work, and cognitive systems engineering and proposes a framework for integrating operational concerns into the concurrent engineering process. Relevance to industry To be competitive, organizations need to effectively manage human and technological resources. A key issue is the nature of the information and technological infrastructure that both enables and supports ‘best practice’ across the enterprise. This paper describes such an approach in the context of the ‘operational enterprise’ and provides both a philosophical stance as well as specific examples of software support.

Toward cooperative advice-giving systems: a case study in knowledge-based decision support

The joint cognitive system approach to problem solving works especially well in creative situations, such as design, that have no single "correct" solution. Our experience in providing expert systems support to a biotechnical equipment manufacturer argues for a shift from classical, problem-solving expert systems to more cooperative, advice-giving systems.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Cooperative problem solving and explanation

Recent studies have pointed out several limitations of expert systems regarding user needs, and have introduced the concepts of cooperation and joint cognitive systems. While research on explanation generation by expert systems has been widely developed, there has been little consideration of explanation in relation to cooperative systems. Our aim is to elaborate a conceptual framework for studying explanation in cooperation. This work relies heavily on the study of human-human cooperative dialogues. We present our results according to two dimensions; namely, the relation between explanation and problem solving, and the explanation process itself. Finally, we discuss the implications of these results for the design of cooperative systems.

The design of joint cognitive systems: the effect of cognitive coupling on performance

In recent years, there has been a growing interdisciplinary interest in designing intelligent human-computer systems for problem-solving. Although progress has been made, we are far from building intelligent human-computer systems that fully exploit the natural synergies of the combination of human and intelligent machine. One of the significant paradigms of intelligent decision support is the cognitive systems engineering approach. This approach considers the human and the intelligent machine as components of a joint cognitive system and focuses on the need to maximize the overall performance of the joint system. Factors influencing the performance of the joint cognitive system include the cognitive characteristics of the human, the computer system, and the task. An important relationship between the cognitive characteristics of the human and those of the system is cognitive coupling, which has a number of dimensions.The study described in this paper explores the style dimension of cognitive coupling by presenting a laboratory experiment that examines the interactions among human cognitive style (analytic vs. heuristic), problem type (analysis-inducing vs. heuristic-inducing), and nature of decision aid (analytic vs. heuristic). The study demonstrates that, depending on the characteristics of the human, the computerized aid, and the problem to be solved, the joint human-computer system performance can be better or worse than the performance of the individual human or computer system working alone. Furthermore, the results suggest that the impact of cognitive style on decision-making performance may depend upon the characteristics of the problem, the nature of the decision-aid, and the measures used to evaluate performance. Inadequate recognition of these factors and their interactions may have led to conflicting results in prior decision-making studies using cognitive style.

SightPlan Experiments: Alternate Strategies for Site Layout Design

SightPlan is a knowledge-based system that lays out temporary facilities on construction sites. Its implementation architecture calls for explicit representation of the problem-solving strategy used to construct a solution layout. Because this strategy can easily be modified we explore several alternate formulations. This paper describes the development and testing of strategies that two agents might adopt to match their cognitive capabilities. One strategy models the steps taken by a construction manager laying out the site; it reflects human cognitive strengths and weaknesses. Recognizing that such a strategy would not be well-suited for execution by a computer, we critique it, and craft a second strategy to make better use of the computer’s storage and computational capabilities. Although more powerful in several ways, this second strategy falls short of some human strengths, such as the ability to discriminate between alternatives. After comparing these two strategies, we develop a third, combining human and machine strengths. This strategy augments the second strategy with graphics to allow for human-computer interaction. We argue that the resulting joint cognitive system can generate better solutions than either a person or a machine.

Intelligent Alarm Handling

The paper outlines the functional structure of an intelligent alarm system. The alarm system should be regarded as a "joint cognitive system" where the operator's need for information at different abstraction levels is taken into account. The alarm system is divided into a low level alarm handling module, a human operator interface and a superior decision support level. The superior level is further decomposed into modules for: event analysis, diagnosis, goal selection, planning and execution. The module for event analysis is subjected to further analysis. We propose that this module should combine information on incoming alarms with other information on the process state available from the process control system. We describe how this may be achieved by using a newly developed method based on the parsing of the incoming event sequence. The parsing process is based on a DCG grammar the describing causal relations in the plant. Parameter passing and calls to Prolog clauses are applied to propagate additional constraints and build a parse tree. The parse tree is used to give a hierar chic description of the actual process state . The levels in the tree correspond to different abstraction levels in the description of the process state.

Intelligent alarm handling

Abstract The paper outlines the functional structure of an intelligent alarm system. The alarm system should be regarded as a “joint cognitive system” where the operator's need for information at different abstraction levels is taken into account. The alarm system is divided into a low level alarm handling module, a human operator interface and a superior decision support level. The superior level is further decomposed into modules for: event analysis, diagnosis, goal selection, planning and execution. The module for event analysis is subjected to further analysis. We propose that this module should combine information on incoming alarms with other information on the process state available from the process control system. We describe how this may be achieved by using a newly developed method based on the parsing of the incoming event sequence. The parsing process is based on a DCG grammar the describing causal relations in the plant. Parameter passing and calls to Prolog clauses are applied to propagate additional constraints and build a parse tree. The parse tree is used to give a hierar chic description of the actual process state. The levels in the tree correspond to different abstraction levels in the description of the process state.

Human interaction with an “intelligent” machine

In this paper we report the results of a study of technicians diagnosing faults in electro-mechanical equipment with the aid of an expert system. Technicians varying in level of experience and interactive style (active or passive) diagnosed faults varying in level of difficulty. The results indicate that the standard approach to expert system design, in which the user is assigned the role of data gatherer for the machine, is inadequate. Problem solving was marked by novel situations outside the machine's conpetence, special conditions, underspecified instructions, and error recovery, all of which required substantial knowledge and active participation on the part of technicians. We argue that the design of intelligent systems should be based on the notion of a joint cognitive system architecture: computational technology should be used to aid the user in the process of solving his problem. The human's role is to achieve total system performance as a manager of knowledge resources that can vary in kind and amount of “intelligence” or power.

Mapping cognitive demands in complex problem-solving worlds

Tool builders have focused, not improperly, on tool building--how to build better performing machine problem-solvers, where the implicit model is a human expert solving a problem in isolation. A critical task then for the designer working in this paradigm is to collect human knowledge for computerization in the stand alone machine problem-solver. But tool use involves more. Building systems that are problem-solvers in isolation does not guarantee high performance in actual work contexts where the performance of the joint person-machine system is the relevant criterion. The key to the effective application of computational technology is to conceive, model, design, and evaluate the joint human-machine cognitive system (Hollnagel & Woods, 1983). Like Gestalt principles in perception, a decision system is not merely the sum of its parts, human and machine. The configuration or organization of the human and machine components is a critical determinant of the performance of the system as a whole (e.g. Sorkin & Woods, 1985). The joint cognitive system paradigm (Woods, 1986; Woods, Roth & Bennett, in press) demands a problemdriven, rather than technology-driven, approach where the requirements and bottlenecks in cognitive task performance drive the development of tools to support the human problem-solver. In this paper we describe an approach to understand the cognitive activities performed by joint human-machine cognitive systems. The real impediment to effective knowledge acquisition is the lack of an adequate language to describe cognitive activities in particular domains--what are the cognitive implications of some application's task demands and of the aids and interfaces available to the people in the system; how do people behave/perform in the cognitive situations defined by these demands and tools. Because this independent cognitive description has been missing, an uneasy mixture of other types of description of a complex situation has been substituted---descriptions in terms of the application itself, of the implementation technology of the interfaces/aids, of the user's physical activities or user psychohaetrics. We describe one approach to provide an independent cognitive description of complex situations that can be used to understand the sources of both good and poor performance, i.e. the cognitive problems to be solved or challenges to be met.