Cognitive Systems Engineering Research Articles

Cognitive Systems Engineering Issues in the Design of Machine Learning Systems

The objective of this panel is to better understand how machine learning (ML) systems influence users and to identify strategies to make such interactions more effective when designing decision support systems and joint cognitive systems for real-time control. The panelists will draw upon their insights from the fields of aviation, defense, ground transportation and medicine, as well as the literature on human-automation and human-AI interaction. The panelists will discuss findings regarding both the positive and negative influences that the design of such systems can have on perceptual, cognitive and decision making processes. They will further provide concrete examples of system designs. Topics for discussion will include: (a) The Ironies of AI Based on Machine Learning: Challenges and New Directions, (b) Understanding the Influences of ML on Joint Cognitive Performance, (c) Mixed Initiative Access to Information, (d) Focusing and Accelerating Decision Making, and (e) Designing the Operational Domain.

Literature review: A study of cognitive aspects in minimizing human error in construction

Human error is a mistake due to human negligence, both consciously and unconsciously, which can cause a system to experience disruption and not run as planned. These errors can occur at the managerial, conceptual, or technical level. One of the activities with a high potential for finding human error is construction activities. Construction activities are a series of activities that include the construction, operation, maintenance, demolition, and construction of a building. Infrastructure development is one of the government’s policy focuses in 2023, with development from four main areas: water resources, roads and bridges, settlements, and housing. These infrastructure projects must have a good quality because they are closely related to user safety and security. Based on data from the Social Security Organizing Agency (BPJS) Employment and statements from the Ministry of PUPR, the industrial sector is an enormous contributor to work accidents with an average percentage of 32% each year. The human factor is one of the forerunners of human error apart from equipment and working environment conditions. Viewed from the perspective of Cognitive System Engineering, human error is caused by the failure of human cognitive processes where the process is related to Human Information Processing (HIP). This paper will describe and briefly discuss the problem of human error and its relation to Human Information Processing (HIP). This paper will examine various literature and summarize efforts that can be applied to improve the information processing process in the hope of minimizing the occurrence of human error in the scope of construction work.

Analysis and Requirement Generation for Defense Intelligence Search: Addressing Data Overload through Human–AI Agent System Design for Ambient Awareness

This research addresses the data overload faced by intelligence searchers in government and defense agencies. The study leverages methods from the Cognitive Systems Engineering (CSE) literature to generate insights into the intelligence search work domain. These insights are applied to a supporting concept and requirements for designing and evaluating a human-AI agent team specifically for intelligence search tasks. Domain analysis reveals the dynamic nature of the ‘value structure’, a term that describes the evolving set of criteria governing the intelligence search process. Additionally, domain insight provides details for search aggregation and conceptual spaces from which the value structure could be efficiently applied for intelligence search. Support system designs that leverage these findings may enable an intelligence searcher to interact with and understand data at more abstract levels to improve task efficiency. Additionally, new system designs can support the searcher by facilitating an ‘Ambient Awareness’ of non-selected objects in a large data field through relevant system cues. Ambient Awareness achieved through the supporting concept and AI teaming has the potential to address the data overload problem while increasing the breadth and depth of search coverage.

Cognitive Systems Engineering at 40, Part I: Deriving Theory from Practice

This expert panel is the first of a two-panel series marking the 40th anniversary of “Cognitive Systems Engineering: New Wine in New Bottles” by Hollnagel and Woods (1983) and, arguably, the beginning of Cognitive Systems Engineering (CSE). These experts were there at (or near) the beginning, devising new methods, expanding and creating new theories, and revealing a new perspective on how complex systems sustain performance and fail. They also wrestled and struggled with these new ideas to propose and implement solutions to improve performance in a number of high-consequence industries. Whether in graduate school or as early-career professionals, they saw the surprises that served as signals that the thinking that brought us to that point would not, alone, be the thinking and doing that would take us further. They will each answer the question, “What ideas and perspectives are important about Cognitive Systems Engineering, and why?”

Cognitive Systems Engineering at 40, Part II: Applying Theory to Practice

This expert panel is the second of a two-panel series marking the 40th anniversary of “Cognitive Systems Engineering: New Wine in New Bottles” by Hollnagel and Woods (1983) and, arguably, the beginning of Cognitive Systems Engineering (CSE). These experts were there at (or near) the beginning. They wrestled with the daunting issues of how to apply new CSE findings, many of which they themselves discovered, into workable solutions that actually improved systems performance in high-uncertainty, high-stakes settings. Their work has been implemented in a number of fields, including aviation, healthcare, power generation, and defense. They will each answer the question, “What ideas and perspectives are important about Cognitive Systems Engineering, and why?”

Towards Joint Activity Design Heuristics: Essentials for Human-Machine Teaming

As machines increasingly behave more like active cognitive agents than passive tools, additional heuristics for supporting joint human-machine activity are urgently needed to complement existing usability heuristics. Despite the rich and extensive design guidance produced by forty years of cognitive systems engineering (CSE) and related fields, the lack of large-scale impact can be attributed, in part, to insufficient translation of CSE principles and guidelines to language and tools that are ready for designers and other decision-makers responsible for these automation-infused solutions. Towards this need, we synthesized a partial and preliminary list of ten machine requirements intended to capture some of the essentials of joint activity. We believe solidifying these essentials and their implications for machines is a first and necessary step towards deriving joint activity design heuristics that are valuable, practical, and sustainable for operational personnel. Through iterative refinement, we believe the combination of strong ideas and strong practicality in these tools can be the basis for a large-scale shift in the design and evaluation of human-machine teams.

Inductive and deductive approaches to modelling of human decision making

This paper presents the two conventional approaches to modelling of human decision making. The inductive approach emphasises the anchoring in empirical data. The deductive approach is based on accepted assumtions from formal decision theory. A comparison of the resulting models show that the outcome is very sirnilar. It is argued that this is because modeiling of decision making basically is a hermeneutical exercise, hence neither purely inductive or deductive. An adequate model of decision making in man-machine systems must be based on a description that captures the complexity of both man and machine, such as Cognitive Systems Engineering. This leads to two essential assumptions about decision making systems, that they must be both causa! and intentional. An alternate cybemetic approach to modeiling of decision making, which includes both of theseassumptions, is briefly described.

Exploring patient flow management through a lens of cognitive systems engineering

Hospitals work to provide quality, safety, and availability to patients with a wide variety of care needs, which makes efficient prioritisation and resource utilisation essential. Anticipation of each patients’ trajectory, while monitoring available resources across the hospital, are major challenges for patient flow management. This study focuses on how hospital patient flow management is realised in situ with the help of concepts from cognitive systems engineering. Five semi-structured interviews with high level managers and shadowing observations of seven full work-shifts with management teams were conducted, to explore how patient flow is coordinated and communicated across the hospital. The data has been analysed using qualitative content analysis. The results describe patient flow management using an adapted Extended Control Model (ECOM) and reveal how authority and information might be better placed closer to clinical work for increased efficiency of patient flow. Practitioner summary: This study describes how a large tertiary paediatric hospital’s patient flow management functions. The results offer a new understanding of how patient flow management is communicated and coordinated across organisational levels of the hospital and how authority and information might be better placed closer to clinical work for increased efficiency.

Toward Joint Activity Design: Augmenting User-Centered Design with Heuristics for Supporting Joint Activity.

From their common roots in Human Factors Engineering, Human-Centered Design and Cognitive Systems Engineering have drifted into distinct fields over the past three decades, each developing beneficial heuristics, design patterns, and evaluation methods for designing for individuals and teams, respectively. GeoHAI, a clinical decision support application for preventing hospital-acquired infection, has yielded positive results in early usability testing and is expected to test positively in supporting joint activity, which will be measured through the novel implementation of Joint Activity Monitoring . The design and implementation of this application provide a demonstration of the possibilities and necessities to unify the work of Human-Centered Design and Cognitive Systems Engineering when designing technologies that are usable and useful to individuals engaged in joint activity with machine counterparts and other people. We are calling this unified process Joint Activity Design, which supports designing for machines to be good team players.

Decision Support for Flexible Manufacturing Systems: Application of the Cognitive Systems Engineering and Ecological Interface Design Approach

We applied the cognitive systems engineering (CSE)/ecological interface design (EID) approach to the work domain of flexible manufacturing systems (FMS). Work domain analyses (WDA) and activity analyses were conducted. Previous concerns regarding the suitability of CSE for FMS are addressed. Principles of EID (i.e., direct perception, direct manipulation, and visual momentum) were applied in designing an ecological interface for a flexible manufacturing system (EcoFlex). The critical links between the products of a WDA and the content of an ecological interface (a key innovation of EID) are made explicit. The evaluations of this interface were very positive across a wide variety of traditional and novel dependent variables (please see the companion manuscript). The overall success of this research program clearly demonstrates that the CSE/EID approach is capable of being applied to FMS. Contributions of this manuscript include both theoretical insights (defining properties of the FMS work domain, concrete examples of the principles of EID, critical links between CSE and EID, resolution of concerns about the overall framework) and practical applications (specific design solutions for FMS and similar work domains, and successful examples of the CSE/EID approach which can be studied and leveraged by students).

Impediments of Cognitive System Engineering in Machine-Human Modeling

A comprehensive understanding of human intelligence is still an ongoing process, i.e., human and information security are not yet perfectly matched. By understanding cognitive processes, designers can design humanized cognitive information systems (CIS). The need for this research is justified because today’s business decision makers are faced with questions they cannot answer in a given amount of time without the use of cognitive information systems. The researchers aim to better strengthen cognitive information systems with more pronounced cognitive thresholds by demonstrating the resilience of cognitive resonant frequencies to reveal possible responses to improve the efficiency of human-computer interaction (HCI). A practice-oriented research approach included research analysis and a review of existing articles to pursue a comparative research model; thereafter, a model development paradigm was used to observe and monitor the progression of CIS during HCI. The scope of our research provides a broader perspective on how different disciplines affect HCI and how human cognitive models can be enhanced to enrich complements. We have identified a significant gap in the current literature on mental processing resulting from a wide range of theory and practice.

Decision Support for Flexible Manufacturing Systems: The Evaluation of an Ecological Interface and Principles of Ecological Interface Design

The cognitive systems engineering (CSE)/ecological interface design (EID) approach was applied in developing decision support for the flexible manufacturing system (FMS) work domain. Four interfaces were designed via the factorial application/non-application of direct perception (DP) and direct manipulation (DM). The capability of these interfaces to support performance in a simulated FMS was evaluated using a variety of traditional and novel dependent variables. The ecological interface (with DP, DM and an intact perception-action loop) provided clearly superior decision support (32 favorable significant results) relative to the other three interfaces (a combined total of 28 favorable significant results). The novel dependent variables were very sensitive. The results are interpreted from three different perspectives: traditional EID, the quality of constraint matching between triadic system components and closed-loop, dynamical control systems. The rationale for an expanded theoretical framework which complements, but does not replace, the original principles of CSE/EID is discussed. The potential for both specific interface features and novel dependent variables to generalize to real-world FMS applications is addressed. The expanded theoretical framework is universally relevant for the development of decision making and problem solving support in all computer-mediated work domains.

A Cognitive Systems Engineering Approach Using Unsupervised Fuzzy C-Means Technique, Exploratory Factor Analysis and Network Analysis-A Preliminary Statistical Investigation of the Bean Counter Profiling Scale Robustness.

A bean counter is defined as an accountant or economist who makes financial decisions for a company or government, especially someone who wants to severely limit the amount of money spent. The rise of the bean counter in both public and private companies has motivated us to develop a Bean Counter Profiling Scale in order to further depict this personality typology in real organizational contexts. Since there are no scales to measure such traits in personnel, we have followed the methodological steps for elaborating the scale’s items from the available qualitative literature and further employed a cognitive systems engineering approach based on statistical architecture, employing cluster, factor and items network analysis to statistically depict the best mathematical design of the scale. The statistical architecture will further employ a hierarchical clustering analysis using the unsupervised fuzzy c-means technique, an exploratory factor analysis and items network analysis technique. The network analysis which employs the use of networks and graph theory is used to depict relations among items and to analyze the structures that emerge from the recurrence of these relations. During this preliminary investigation, all statistical techniques employed yielded a six-element structural architecture of the 68 items of the Bean Counter Profiling Scale. This research represents one of the first scale validation studies employing the fuzzy c-means technique along with a factor analysis comparative design.

Metrics for Human-Robot Team Design: A Teamwork Perspective on Evaluation of Human-Robot Teams

Metrics for human-robot teaming should extend to teams consisting of multiple human and robotic agents, and to teams working in complex, dynamic work domains. This work proposes that to comprehensively analyze and evaluate multi-human, multi-robot teams, traditional HRI metrics of performance, and efficiency must be expanded upon to incorporate metrics of teamwork. We develop five distinct metrics to capture both ecological and cognitive aspects of teamwork found to be important in human-automation interaction, inspired by research in the cognitive systems engineering (CSE) community. We demonstrate the application of these metrics in a spacecraft maintenance case study comparing multiple human-robot team architectures. The case study demonstrates that the teamwork metrics capture aspects of human-robot interaction (HRI) not apparent when using only traditional performance and efficiency metrics. The article concludes that the proposed teamwork metrics are complementary to existing metrics in HRI and should be included in the evaluation of human-robot teams.

Challenges from the Introduction of Artificial Intelligence in the European Air Traffic Management System

The Air Traffic Management (ATM) system can be defined as a “Joint Cognitive System” of people, teams, and artifacts that adapts to the challenges and demands posed by familiar and unfamiliar situations in a dynamically evolving operational context. In the era of digitalization and Big Data we live, an incremental modernization of the ATM system is expected in the coming years with the pervasive implementation of Artificial Intelligence (AI) and Machine Learning (ML). In this paper, we present the findings from an initial attempt to detect and document the fundamental challenges of the introduction of AI, in the European ATM system through the lens of Cognitive Systems Engineering paradigm. We also discuss how these challenges give rise to difficult to resolve safety and performance related patterns in the ATM system.

Online Instruction for Cognitive Systems Engineering: Lessons Learned and Opportunities to Pursue

One of the impacts of the pandemic has been a rapid increase in the development and offering of online courses focused on cognitive systems engineering. This presents opportunities to: ● Identify and share alternative instructional design strategies and more specific instructional tactics tailored to the online environment and learn from each others’ experiences. ● Discuss how lessons learned from the design and offering of online courses can not only inform future offerings of online courses but also generalize to in-person courses. ● Identify the opportunities created by online instruction to reach a broader audience, not only geographically but also in terms of reaching practitioners whose specializations are outside of human factors. The panelists have expertise and experience in all of these areas. Their perspectives are briefly described below.

RETRACTED ARTICLE: Application of cognitive system engineering in development of dragon boat sports

Dragon boat is a traditional sport in China. A trajectory control model of dragon boat movement development based on cognitive system engineering is proposed to achieve quantitative planning and cognitive design of dragon boat movement development. The constraint parameter model of the evolution control which affects the development of the dragon boat movement is constructed, and the state optimization and adaptive game equilibrium design of the development of the dragon boat movement are carried out by using the game control method, and the cognitive structure of the development of the dragon boat movement is combined. In order to improve the cognitive control ability of the development of the dragon boat, the cognitive system engineering design of the development of the dragon boat movement is carried out, and the methods of small disturbance suppression and adaptive equilibrium configuration are adopted to recognize the pattern and plan the state of the development process of the dragon boat movement. The cognitive system engineering design and control optimization for the development of dragon boat motion are realized by using joint parameter estimation and adaptive search control method. The simulation results shows that it is of better self-adaptability using this method to launch the cognitive system control and planning of dragon boat sports development, and the ability of self-adaptive process control and system cognition on dragon boat sports development has been improved.

Why is the Implementation of Crew Resource Management through Checklists in Healthcare not Systematically Successful? An Answer from Cognitive Systems Engineering

Why is the Implementation of Crew Resource Management through Checklists in Healthcare not Systematically Successful? An Answer from Cognitive Systems Engineering

Cognitive Work of Hypothesis Exploration During Anomaly Response

Four incidents from web-based software companies reveal important aspects of anomaly response processes when incidents arise in web operations, two of which are discussed in this article. One particular cognitive function examined in detail is hypothesis generation and exploration, given the impact of obscure automation on engineers’ development of coherent models of the systems they manage. Each case was analyzed using the techniques and concepts of cognitive systems engineering. The set of cases provides a window into the cognitive work "above the line" in incident management of complex web-operation systems.

Cognitive Systems Engineering Applied to Decision Support in Space Situational Awareness

Existing approaches for sensor network tasking in space situational awareness (SSA) rely on techniques from the 1950s and limited application areas while also requiring significant human-in-the-loop involvement. Increasing numbers of space objects, sensors, and decision-making needs create a demand for improved methods of gathering and fusing disparate information to resolve hypotheses about the space object environment. This work focuses on the cognitive work in SSA sensor tasking approaches. The application of a cognitive work analysis for the SSA domain highlights capabilities and constraints inherent to the domain that can drive SSA operations toward decision-maker goals. A control task analysis is also conducted to derive requirements for cognitive work and information relationships that support the information fusion and sensor allocation tasks of SSA. A prototype decision-support system is developed using a subset of the derived requirements. This prototype is evaluated in a human-in-the-loop experiment using both a hypothesis-based and covariance-based scheduling approaches. Results from this preliminary evaluation show operator ability to address SSA decision-maker hypotheses using the prototype decision-support system (DSS) using both scheduling approaches.