Joint Cognitive Systems Research Articles

Cognitive Systems Engineering Issues in the Design of Machine Learning Systems

The objective of this panel is to better understand how machine learning (ML) systems influence users and to identify strategies to make such interactions more effective when designing decision support systems and joint cognitive systems for real-time control. The panelists will draw upon their insights from the fields of aviation, defense, ground transportation and medicine, as well as the literature on human-automation and human-AI interaction. The panelists will discuss findings regarding both the positive and negative influences that the design of such systems can have on perceptual, cognitive and decision making processes. They will further provide concrete examples of system designs. Topics for discussion will include: (a) The Ironies of AI Based on Machine Learning: Challenges and New Directions, (b) Understanding the Influences of ML on Joint Cognitive Performance, (c) Mixed Initiative Access to Information, (d) Focusing and Accelerating Decision Making, and (e) Designing the Operational Domain.

Joint Cognitive Systems: Ideas With Impact for Designing Safer, More Resilient Automated Systems

Joint Cognitive Systems: Ideas With Impact for Designing Safer, More Resilient Automated Systems

Applying HCAI in Developing Effective Human-AI Teaming: A Perspective from Human-AI Joint Cognitive Systems

Applying HCAI in Developing Effective Human-AI Teaming: A Perspective from Human-AI Joint Cognitive Systems

Designing human-AI systems for complex settings: ideas from distributed, joint, and self-organising perspectives of sociotechnical systems and cognitive work analysis

Real-world events like the COVID-19 pandemic and wildfires in Australia, Europe, and America remind us that the demands of complex operational settings are met by multiple, distributed teams interwoven with a large array of artefacts and networked technologies, including automation. Yet, current models of human-automation interaction, including those intended for human-machine teaming or collaboration, tend to be dyadic in nature, assuming individual humans interacting with individual machines. Given the opportunities and challenges of emerging artificial intelligence (AI) technologies, and the growing interest of many organisations in utilising these technologies in complex operations, we suggest turning to contemporary perspectives of sociotechnical systems for a way forward. We show how ideas of distributed cognition, joint cognitive systems, and self-organisation lead to specific concepts for designing human-AI systems, and propose that design frameworks informed by contemporary views of complex work performance are needed. We discuss cognitive work analysis as an example.

Spatially-aware group interaction design framework for collaborative room-oriented immersive systems

Room-oriented immersive systems are human-scale built environments that enable collective multi-sensory immersion in virtual space. Although such systems are currently seeing increasing applications in public realms, limited understanding remains regarding how humans interact with the virtual environments displayed within. Synthesizing virtual reality ergonomics and human-building interaction (HBI) knowledge allows us to investigate these systems meaningfully. In this work, we develop a model of content analysis based on hardware components of the Collaborative-Research Augmented Immersive Virtual Environment Laboratory (CRAIVE-Lab) and the Cognitive Immersive Room (CIR) at Rensselaer Polytechnic Institute. Situating ROIS as a joint cognitive system, this model consists of five categories of qualitative factors: 1) general design approach; 2) topological relationships; 3) features of tasks; 4) hardware-specific design modalities; and 5) interactive qualities. We probe the comprehensiveness of this model using existing design scenarios at the CRAIVE-Lab and the CIR featuring both application-based and experience-based designs. Through these case studies, the robustness of this model in its representation of design intention is observed, with limitations on temporal constraints. In creating this model, we establish foundations for more detailed assessments of the interactive qualities of systems alike.

Interdependence Analysis in collaborative robot applications from a joint cognitive functional perspective

This paper sets up a framework to assess co-agency in human-robot interactions, and applies it specifically to the socio-technical safety analysis of collaborative robots. We also examine to what extent the concept of Situation Awareness can be applied to assess collaborative robots as efficient team members in socio-technical systems. We explain some theoretical concerns with traditional concepts of Situation Awareness and defend why the concept of Joint Cognitive Systems, which maps the conceptualization of the cognitive system onto the work system as a whole, is best suited for issues of distributed cognition and controllability in human-robot interaction. Thereafter we present a five-step methodology specifically conceived for cobot applications serving the aim of goal coordination between multiple agents by functional interactions. The proposed framework merges two existing safety and resilience analysis methods, being the Functional Resonance Analysis Method and Interdependence Analysis. These methods are used in combination to assess shared control in safe and efficient human-robot interaction from a systems-thinking perspective. This allows to describe the systemic conditions for Distributed Situation Awareness in terms of observable system interactions and as an emergent object of distributed cognition. Instead of looking at undesirable safety outcomes, we have imposed the focus of co-agency as the unit of analysis in line with the Joint Cognitive Systems perspective. The theoretical insights from this paper are additionally applied to a hypothetical but credible demonstration case study with collaborative warehouse robots.

Smooth and Resilient Human–Machine Teamwork as an Industry 5.0 Design Challenge

Smart machine companions such as artificial intelligence (AI) assistants and collaborative robots are rapidly populating the factory floor. Future factory floor workers will work in teams that include both human co-workers and smart machine actors. The visions of Industry 5.0 describe sustainable, resilient, and human-centered future factories that will require smart and resilient capabilities both from next-generation manufacturing systems and human operators. What kinds of approaches can help design these kinds of resilient human–machine teams and collaborations within them? In this paper, we analyze this design challenge, and we propose basing the design on the joint cognitive systems approach. The established joint cognitive systems approach can be complemented with approaches that support human centricity in the early phases of design, as well as in the development of continuously co-evolving human–machine teams. We propose approaches to observing and analyzing the collaboration in human–machine teams, developing the concept of operations with relevant stakeholders, and including ethical aspects in the design and development. We base our work on the joint cognitive systems approach and propose complementary approaches and methods, namely: actor–network theory, the concept of operations and ethically aware design. We identify their possibilities and challenges in designing and developing smooth human–machine teams for Industry 5.0 manufacturing systems.

Challenges from the Introduction of Artificial Intelligence in the European Air Traffic Management System

The Air Traffic Management (ATM) system can be defined as a “Joint Cognitive System” of people, teams, and artifacts that adapts to the challenges and demands posed by familiar and unfamiliar situations in a dynamically evolving operational context. In the era of digitalization and Big Data we live, an incremental modernization of the ATM system is expected in the coming years with the pervasive implementation of Artificial Intelligence (AI) and Machine Learning (ML). In this paper, we present the findings from an initial attempt to detect and document the fundamental challenges of the introduction of AI, in the European ATM system through the lens of Cognitive Systems Engineering paradigm. We also discuss how these challenges give rise to difficult to resolve safety and performance related patterns in the ATM system.

JOINT COGNITIVE SYSTEM IN THE CONTEXT OF DIGITALIZATION OF PROFESSIONAL EDUCATION

The article deals with the risks of digitalization of professional education, and describes the organization and func­ tioning of the joint cognitive systems «teacher–educational information–student» in the educational process of the university based on the key positions of a teacher and the age characteristics of a modern student.

Развитие культуры интеллектуальной деятельности студента на основе принципов когнитивной эргономики

The versatility of the concept and the need to develop the culture of students’ intellectual activity as a way of adapting in the modern information space are shown. The article considers the features of designing a joint cognitive system “The teacher – educational information – the student” as a condition for developing the culture of a future specialist’s intellectual activity and improving the quality of mastering educational information in the professional training process. The results of studying the factors reducing the quality of a student’s intellectual and educational activities within higher education environment are analyzed and discussed. Findings and recommendations are made to direct psychological and pedagogical correction in the system of higher education for further research.

Teaming with industrial cobots: A socio‐technical perspective on safety analysis

AbstractCollaborative human–machine interaction will be progressively intensified in industrial applications. The aim of this article is to examine current approaches to cobot safety by showing that these approaches can additionally benefit from systems thinking methods. The first part of this article covers a narrative literature review on predominantly techno‐centric robot safety approaches, with a strong focus on containing kinetic energy and ensuring separation with humans. The second part introduces systems thinking methods to analyze a socio‐technical perspective on cobot safety, including joint cognitive systems and distributed cognition perspectives. This explorative research dimension is expected to overcome an overly narrow interpretation of safety issues, anticipating the challenges ahead in ever more complex cobot applications. This article embraces a socio‐technical perspective to explore the potential of Joint Cognitive Systems to manage risk and safety in cobot applications. Three systemic safety analysis approaches are presented and tested with a demonstrator case study concerning their feasibility for cobot applications: System‐Theoretic Accident Model and Processes (STAMP); Functional Resonance Analysis Method (FRAM); and Event Analysis of Systemic Teamwork (EAST). These methods each provide interesting extensions to complement the traditional understanding of risk as required by current and future industrial cobot implementations. The power of systemic methods for safer and more efficient cobot operations lies in revealing the distributed and emergent result from joint actions and overcoming the reductionist view from individual failures or single agent responsibilities. The safe operation of cobot applications can only be achieved through alignment of design, training, and operation of such applications.

A head mounted augmented reality design practice for maintenance assembly: Toward meeting perceptual and cognitive needs of AR users

Head Mounted Display (HMD) based Augmented Reality (AR) is being increasingly used in manufacturing and maintenance. However, limited research has been done to understand user interaction with AR interfaces, which may lead to poor usability, risk of occupational hazards, and low acceptance of AR systems. This paper uses a theoretically-driven approach to interaction design to investigate the impact of different AR modalities in terms of information mode (i.e. video vs. 3D animation) and interaction modality (i.e. hand-gesture vs. voice command) on user performance, workload, eye gaze behaviours, and usability during a maintenance assembly task. The results show that different information modes have distinct impacts compared to paper-based maintenance, in particular, 3D animation led to a 14% improvement over the video instructions in task completion time. Moreover, insights from eye gaze behaviours such as number of fixations and transition between Areas of Interest (AOIs) revealed the differences in attention switching and task comprehension difficulty with the choice of AR modalities. While, subjective user perceptions highlight some ergonomic issues such as misguidance and overreliance, which must be considered and addressed from the joint cognitive systems' (JCSs) perspective and in line with the predictions derived from the Multiple Resources Model.

Requirements for Computational Approaches to Analyzing Resilience in Human-Machine Teams

Human-machine teams (HMTs) in complex work domains need to be able to adapt to variable and uncertain work demands. Computational modeling and simulation can provide novel approaches to the evaluation of HMTs performing complex joint activities, affording large-scale, quantitative analysis of team characteristics (such as system architecture and governance protocols) and their effects on resilience. Drawing from literature in resilience engineering, human-automation interaction, and cognitive systems engineering, this paper provides a theoretical exploration of the use of computational modeling and simulation to analyze resilience in HMTs. Findings from literature are summarized in a set of requirements that highlight key aspects of resilience in HMTs that need to be accounted for in future modeling and evaluation efforts. These requirements include a need to model HMTs as joint cognitive systems, the need to account for the interdependent nature of activity, the temporal dynamics of work, and the need to support formative exploration and inquiry. We provide a brief overview of existing modeling and simulation approaches to evaluating HMTs and discuss further steps for operationalizing the identified requirements.

Modeling the Effects of Machine Rigidities on Joint Work Strategies

One of the challenges in designing resilient human-machine systems is that machine capabilities are inherently rigid. A resilient joint cognitive system can anticipate and adapt to changing work demands effectively, but limitations of machines can make this adaptation constrained and less fluid. By identifying and accommodating for these rigidities in the design of human-machine system architectures, developers can build human-machine systems that support multiple contexts. This paper proposes a work-modeling approach for analyzing joint human-machine work strategies, focusing on identifying interdependencies that would support opportunistic adaptation and reduce the risk of machine rigidity leading to brittle failures of a human-machine system. The approach is applied to a case study in space operations to demonstrate how interdependencies can be identified and evaluated. The results of this analysis provide early insight into how team adaptation and machine limitations can be systematically accounted for in system architecture design.

Mastering Automation: New Airline Pilots’ Perspective

ABSTRACT Aircraft accidents over the last decade have raised questions about the most effective way to transition pilots onto new aircraft automation systems. This debate can range from the details contained within aircraft flight manuals to computer-based versus simulator-based training. While some questions are still under consideration, most fall short in the understanding of how pilots “actually” learn. The researchers conducted a longitudinal study focusing on pilot learning; through interviews, observations, and field documentation, 10 pilot trainees from two regional airlines were followed for 5 months throughout their initial airline training. Using joint cognitive systems theory, and concepts from curriculum theory to understand the utility of training programs, the study revealed gaps between the airlines’ intended learning outcomes and the trainees’ actual experiences. The findings demonstrate that automation, as opposed to other aircraft systems, may require different approaches to learning. Recommendations for possible modifications to current pilot training are discussed.

Practitioner perspectives on the implementation of an electronic information system to enforce practice standards in England

ABSTRACT Research in England, mainland Europe and Australia has demonstrated how the introduction of electronic information systems (IS) has been highly problematic for frontline social workers, particularly in terms of diverting their time, attention and energy away from direct work with service users. IS have evolved over the years to become increasingly complex and demanding in terms of the nature and volume of data that is required to be entered by social workers. While incorporating workflows in an IS that direct social workers to specific tasks at a particular time is not new, one recently implemented IS incorporates a plethora of workflows that aim to enforce compliance with practice procedures and standards. In this article, the findings from a small qualitative study which aimed to evaluate the reaction of social workers to this new form of IS are reported. Going beyond the problems expected with the introduction of a new IS and drawing theoretically from social informatics and joint cognitive systems, the reasons why this form of IS may be particularly problematic for frontline social workers are explored.

Evaluating Impacts of Head Worn Displays on Teamwork in Emergency Response: Review of Challenges for the Field

Head Worn Displays (HWDs) are increasingly used to support mobile workers across diverse domains. However, little is known about how HWDs affect teamwork in safety-critical contexts. We conducted a narrative review examining the effects of HWDs on teamwork performance and team processes of situation awareness, communication, and coordination for emergency response. HWDs appear to improve the quality of team performance but increase time to perform under some conditions; effects on team processes are also mixed. It is difficult to compare results across studies due to the diverse technologies, contexts, and measures used. Successful design, evaluation, and deployment of HWDs in emergency response contexts may require a stronger foundation of theory-driven and process-based research. Perspectives such as joint cognitive systems, distributed cognition, and common ground may help researchers uncover mechanisms by which HWDs shape teamwork processes and how team processes affect team performance over time.

Analysis of the cognitive demands of electronic health record use

The goal of this study was to elicit the cognitive demands facing clinicians when using an electronic health record (EHR) system and learn the cues and strategies expert clinicians rely on to manage those demands. This study differs from prior research by applying a joint cognitive systems perspective to examining the cognitive aspects of clinical work. We used a cognitive task analysis (CTA) method specifically tailored to elicit the cognitive demands of an EHR system from expert clinicians from different sites in a variety of inpatient and outpatient roles. The analysis of the interviews revealed 145 unique cognitive demands of using an EHR, which were organized into 22 distinct themes across seven broad categories. In addition to confirming previously published themes of cognitive demands, the main emergent themes of this study are: 1) The EHR does not help clinicians develop and maintain awareness of the big picture; 2) The EHR does not support clinicians’ need to reason about patients’ current and future states, including effects of potential treatments; and 3) The EHR limits agency of clinicians to work individually and collaboratively. Implications for theory and EHR design and evaluation are discussed.

Getting mixed signals: Connotations of teamwork as performance shaping factors in network controller and rail driver relationship dynamics

Signal passed at danger (SPAD) events have been a perennial issue in rail and requires continued scholarship. Much of the literature has tended to focus on activities within the train cab, placing the spotlight on “error” within the rail driving role; however a train is not propelled by a single person but by a team. This study set out to understand how network controllers perceive the relationship dynamic between the network controller and rail driver, and how these views shape system performance. Using a phenomenological approach underpinned by Joint Cognitive Systems theory, interviews were conducted with network controllers (N = 55) across 8 rail organisations in Australia and New Zealand to examine how they related to signals and to drivers. Sixteen different perspectives were identified ranging in type and varying by frequency, each with implications for coupling strength between the controller and driver roles and on system performance. Key dimensions of teamwork in train movement were emphasised, illustrating how the underlying values and philosophies in different network controlling cultures influence risk perception. The findings may be used to develop a more informed understanding of the ‘controller-signal-driver-train’ system and pave the way for strategies that can embrace diversity in different perspectives whilst optimizing system safety and performance.

A socio-technical analysis of functional properties in a joint cognitive system: a case study in an aircraft cockpit

In a socio-technical work domain, humans, device interfaces and artefacts all affect transformations of information flow. Such transformations, which may involve a change of auditory to visual information & vice versa or alter semantic approximations into spatial proximities from instruments readings, are generally not restricted to solely human cognition. This paper applies a joint cognitive system approach to explore a socio-technical system. A systems ergonomics perspective is achieved by applying a multi-layered division to transformations of information between, and within, human and technical agents. The approach uses the Functional Resonance Analysis Method (FRAM), but abandons the traditional boundary between medium and agent in favour of accepting aircraft systems and artefacts as agents, with their own functional properties and relationships. The joint cognitive system perspective in developing the FRAM model allows an understanding of the effects of task and information propagation, and eventual distributed criticalities, taking advantage of the functional properties of the system, as described in a case study related to the cockpit environment of a DC-9 aircraft. Practitioner Summary: This research presents the application of one systemic method to understand work systems and performance variability in relation to the transformation of information within a flight deck for a specific phase of flight. By using a joint cognitive systems approach both retrospective and prospective investigation of cockpit challenges will be better understood. Abbreviations: ATC: air traffic control; ATCO: air traffic controller; ATM: air traffic management; CSE: cognitive systems engineering; DSA: distributed situation awareness; FMS: flight management system; FMV: FRAM model visualize; FRAM: functional resonance analysis method; GF: generalised function; GW: gross weight; HFACS: human factors analysis and classification system; JCS: joint cognitive systems; PF: pilot flying; PNF: pilot not flying; SA: situation awareness; SME: subject matter expert; STAMP: systems theoretic accident model and processes; VBA: visual basic for applications; WAD: work-as-done; WAI: work-as-imagined; ZFW: zero fuel weight