Cognitive Engineering Methods Research Articles

Where the Cognitive Engineering Rubber Meets the Systems Development Road

The fields of Cognitive Engineering (CE) and Decision Making (DM) have made exceptional contributions to our understanding of complex sociotechnical systems; however, it can be difficult to apply research and findings from CE work to drive Systems Development (SD). This panel discussion examines the connective tissue between the worlds of CE research and SD practice. Panelists with backgrounds across CE and SD will provide generalizable examples of how to apply specific CE theories, methods, and principles (e.g. naturalistic decision making, macrocognition, and resilience engineering), to generate various SD products (e.g. system requirements and designs). The goal of the panel will be to provide useful examples, and spur discussion about means to more readily translate research into practice.

Using the Technology Acceptance Model to Develop StartSmart: mHealth for Screening, Brief Intervention, and Referral for Risk and Protective Factors in Pregnancy

Technology decision support with tailored patient education has the potential to improve maternal and child health outcomes. The purpose of this study was to develop StartSmart, a mobile health (mHealth) intervention to support evidence-based prenatal screening, brief intervention, and referral to treatment for risk and protective factors in pregnancy. StartSmart was developed using Davis' Technology Acceptance Model with end users engaged in the technology development from initial concept to clinical testing. The prototype was developed based upon the current guidelines, focus group findings, and consultation with patient and provider experts. The prototype was then alpha tested by clinicians and patients. Clinicians were asked to give feedback on the screening questions, treatment, brief motivational interviewing, referral algorithms, and the individualized education materials. Clinicians were asked about the feasibility of using the materials to provide brief intervention or referral to treatment. Patients were interviewed using the think aloud technique, a cognitive engineering method used to inform the design of mHealth interventions. Interview questions were guided by the Screening, Brief Intervention, Referral to Treatment theory and attention to usefulness and usability. Expert clinicians provided guidance on the screening instruments, resources, and practice guidelines. Clinicians suggested identifying specific prenatal visits for the screening (first prenatal visit, 28-week visit, and 36-week visit). Patients reported that the tablet-based screening was useful to promote adherence to guidelines and provided suggestions for improvement including more information on the diabetic diet and more resources for diabetes. During alpha testing, participants commented on navigability and usability. Patients reported favorable responses about question wording and ease of use. Clinicians reported the use of mHealth to screen and counsel pregnant patients on risk and protective factors facilitated their ability to provide comprehensive care.

Psychosocial and cognitive engineering contributions to blood transfusion

Among the research developed in social psychology, there is a field of study questioning the mechanisms that explain prosocial behaviors. In 1956, the first links between transfusion medicine and social psychology were formed, with the study of underlying motivations to blood donation behavior. Sixty years later, the number of “fundamental” and “applied” research has widely increased, and we now have a theoretical understanding of blood donation engagement's levers, as well as experimental demonstrations of methods to activate these levers. Thus, the literature offers several strategies experimentally verified to improve the recruitment and retaining of blood donors. These methods can go from specific speeches when soliciting donors, to changes in the blood donation environment, etc. They aim either at impacting the donation experience, at obtaining a change in the donor's behavior, or at changing the determinants of blood donation. Yet, it is clear that the knowledge from the research in social psychology is not really deployed on the field by the organizations that could benefit from it. In this article, we will try to develop the difficulty regarding the application of fundamental and experimental knowledge, and to underline the implications for the blood transfusion establishments. We will then present the psychosocial and cognitive engineering method, and argue its relevance to answer this problematic.

Improving manufacturing productivity by combining cognitive engineering and lean-six sigma methods.

Increasing automation of manufacturing has changed the relationship between workers and the technology employed to get work done. Cognitive work is replacing manual labour. Established productivity improvement methods need to comprehend increased complexity, less tractable processes and a workforce where expertise is a differentiator of outcomes. This paper describes how human factors and cognitive engineering methods can be combined with Lean - Six Sigma to create an enhanced productivity framework for complex manufacturing. Action research over a three-month period is used to validate its effectiveness recording the cognitive work of engineers engaged in a quality control task.

A Digital Coach That Provides Affective and Social Learning Support to Low-Literate Learners

In this study, we investigate if a digital coach for low-literate learners that provides cognitive learning support based on scaffolding can be improved by adding affective learning support based on motivational interviewing, and social learning support based on small talk. Several knowledge gaps are identified: motivational interviewing and small talk must be translated to control rules for this coach, a formal model of participant emotional states is needed to allow the coach to parse the learner's emotional state, and various sensors must be used to let the coach detect and act on this state. We use the situated Cognitive Engineering (sCE) method to update an existing foundation of knowledge with emotional models, motivational interviewing, and small talk theory, technology, and a new exercise in the volunteer work domain. We use this foundation to create a design specification for an Embodied Conversational Agent (ECA) coach that provides cognitive, affective, and social learning support for this exercise. A prototype is created, and compared to a prototype that only provides cognitive support in a within- and between-subjects experiment. Results show that both prototypes work as expected: learners interact with the coach and complete all exercises. Almost no significant differences are found between the two prototypes, indicating that the affective and social support were not effective as designed. Potential improvements are provided for future work. Results also show significant differences between two subgroups of low-literate participants, and between men and women, reinforcing the importance of using individualized support measures with this demographic.

Improving How We Weave Cognitive Engineering into Military Acquisition: Understand, Design, Validate

To fulfill its name and purpose, cognitive engineering must emerge from its application primarily as a research practice into implementation within recognized practices of systems engineering. Although generally accepted in military research circles, cognitive engineering theory and methods have not gained widespread application in systems engineering processes. This panel will enable five speakers with different perspectives on this problem to describe means for attempting to change this state of the practice. The focus of the panel will be in gaining traction and advocacy for applying cognitive engineering in the acquisition of major military systems, which are typically employed in high-stakes, time-critical, high-uncertainty environments.

Low-literates’ support needs for societal participation learning: Empirical grounding of theory- and model-based design

Specialized learning support software can address the low societal participation of low-literate Dutch citizens. We use the situated Cognitive Engineering method to iteratively create a design specification for the envisioned system VESSEL: a Virtual Environment to Support the Societal participation Education of Low-literates. An initial high-level specification for this system is refined by incorporating the societal participation experiences of low-literate citizens into the design. In two series of user studies, the participant workshop and cultural probe methods were used with 23 low-literate participants. The Grounded Theory method was used to process the rich user data from these studies into the Societal Participation Experience of Low-Literates (SPELL) model. Using this experience model, the existing VESSEL specification was refined: requirements were empirically situated in the daily practice of low-literate societal participation, and new claims were written to explicate the learning effectiveness of the proposed VESSEL system. In conclusion, this study provides a comprehensive, theoretically and empirically grounded set of requirements and claims for the proposed VESSEL system, as well as the underlying SPELL model, which captures the societal participation experiences of low-literates citizens. The research methods used in this study are shown to be effective for requirements engineering with low-literate users.

Cognitive engineering and health informatics: Applications and intersections

Cognitive engineering is an applied field with roots in both cognitive science and engineering that has been used to support design of information displays, decision support, human-automation interaction, and training in numerous high risk domains ranging from nuclear power plant control to transportation and defense systems. Cognitive engineering provides a set of structured, analytic methods for data collection and analysis that intersect with and complement methods of Cognitive Informatics. These methods support discovery of aspects of the work that make performance challenging, as well as the knowledge, skills, and strategies that experts use to meet those challenges. Importantly, cognitive engineering methods provide novel representations that highlight the inherent complexities of the work domain and traceable links between the results of cognitive analyses and actionable design requirements. This article provides an overview of relevant cognitive engineering methods, and illustrates how they have been applied to the design of health information technology (HIT) systems. Additionally, although cognitive engineering methods have been applied in the design of user-centered informatics systems, methods drawn from informatics are not typically incorporated into a cognitive engineering analysis. This article presents a discussion regarding ways in which data-rich methods can inform cognitive engineering.

Requirements for a virtual environment to support the social participation education of low-literates

People of low literacy experience difficulties while participating in society. Learning support software could help alleviate these difficulties. However, there is currently no overview of theoretically and empirically sound requirements for this kind of support. This paper uses the situated cognitive engineering method to create a requirements baseline for a virtual environment to support the societal participation education of low-literates (VESSEL), based on an analysis of the domain, human factors, and current applications. Four major outcomes are presented. First, a comprehensive overview is collected of the operational demands and human factors knowledge relevant to societal participation learning for low-literate citizens. Second, this overview is translated into a list of eight functional requirements: focused on low-literate learners, set in the context of societal participation, and supported by claims of cognitive, affective, and social benefits to learning. Third, a sample of Dutch societal participation learning support programs is assessed using these requirements, to highlight both current technology best practices and discrepancies between theory and practice. Fourth, virtual learning environment technology is suggested as an ‘enabling’ technology; an overview is shown of how virtual environments, actors, and objects can beneficially enable meeting the requirements baseline. Finally, directions for future study are discussed.

The Role of Human Factors in Requirements Engineering in Health Care: A Case Study in the Brazilianhealth Care System

In this paper we propose an approach for the enhancement of software requirements specifications by combining human factors and requirements engineering concepts. The approach we propose uses cognitive engineering methods to increase understanding about how professionals perform complex work, taking into account the cognitive effort made by workers. We employed the suggested the approach in the context of a primary health care facility in Brazil to demonstrate that including human factors models improved the understanding of context, decision making and informal knowledge, enhancing software requirements specifications. Our results point out that a richer set of requirements was obtained, increasing the reliability of requirements specifications, enabling the design of better information technology to support work in the primary health care facility.

Advancing cognitive engineering methods to support user interface design for electronic health records

BackgroundDespite many decades of research on the effective development of clinical systems in medicine, the adoption of health information technology to improve patient care continues to be slow, especially in ambulatory settings. This applies to dentistry as well, a primary care discipline with approximately 137,000 practitioners in the United States. A critical reason for slow adoption is the poor usability of clinical systems, which makes it difficult for providers to navigate through the information and obtain an integrated view of patient data. ObjectiveIn this study, we documented the cognitive processes and information management strategies used by dentists during a typical patient examination. The results will inform the design of a novel electronic dental record interface. MethodsWe conducted a cognitive task analysis (CTA) study to observe ten general dentists (five general dentists and five general dental faculty members, each with more than two years of clinical experience) examining three simulated patient cases using a think-aloud protocol. ResultsDentists first reviewed the patient's demographics, chief complaint, medical history and dental history to determine the general status of the patient. Subsequently, they proceeded to examine the patient's intraoral status using radiographs, intraoral images, hard tissue and periodontal tissue information. The results also identified dentists’ patterns of navigation through patient's information and additional information needs during a typical clinician–patient encounter. ConclusionThis study reinforced the significance of applying cognitive engineering methods to inform the design of a clinical system. Second, applying CTA to a scenario closely simulating an actual patient encounter helped with capturing participants’ knowledge states and decision-making when diagnosing and treating a patient. The resultant knowledge of dentists’ patterns of information retrieval and review will significantly contribute to designing flexible and task-appropriate information presentation in electronic dental records.

Better Pairing of Providers and Tools

Health IT systems are often designed without a sufficient understanding of the clinical activities they are intended to support; thus desired benefits in quality of care, safety, and efficiency may not accrue. We present the second part of a multi-phase study which utilizes cognitive systems engineering (CSE) methods to design and test novel user interfaces for a hospital emergency department (ED) information system. Using the work domain analysis results from phase one, display components were developed for a system which can better support work activities and information needs of ED clinicians and staff.

Where’s the Beef: How Pervasive Is Cognitive Engineering in Military Research & Development Today?

Cognitive Engineering methods were developed to enable human factors practitioners to understand and systematically support the cognitive work of people working “at the sharp end of the spear.” Military members for whom DoD acquisition organizations develop systems are the quintessential “sharp end of the spear.” This panel is proposed to share present-day experience from military and industry reflecting how pervasively Cognitive Engineering is contributing to research and development for the highly complex military systems being operated under conditions of stress, time pressure, and uncertainty today. The implications for human factors practitioners will be highlighted, both in terms of practices to continue and areas for improvement.

Comparison of alphabetical versus categorical display format for medication order entry in a simulated touch screen anesthesia information management system: an experiment in clinician-computer interaction in anesthesia

BackgroundAnesthesia information management system (AIMS) records should be designed and configured to facilitate the accurate and prompt recording of multiple drugs administered coincidentally or in rapid succession.MethodsWe proposed two touch-screen display formats for use with our department’s new EPIC touch-screen AIMS. In one format, medication “buttons” were arranged in alphabetical order (i.e. A-C, D-H etc.). In the other, buttons were arranged in categories (Common, Fluids, Cardiovascular, Coagulation etc.). Both formats were modeled on an iPad screen to resemble the AIMS interface. Anesthesia residents, anesthesiologists, and Certified Registered Nurse Anesthetists (n = 60) were then asked to find and touch the correct buttons for a series of medications whose names were displayed to the side of the entry screen. The number of entries made within 2 minutes was recorded. This was done 3 times for each format, with the 1st format chosen randomly. Data were analyzed from the third trials with each format to minimize differences in learning.ResultsThe categorical format had a mean of 5.6 more drugs entered using the categorical method in two minutes than the alphabetical format (95% confidence interval [CI] 4.5 to 6.8, P < 0.0001). The findings were the same regardless of the order of testing (i.e. alphabetical-categorical vs. categorical - alphabetical) and participants’ years of clinical experience. Most anesthesia providers made no (0) errors for most trials (N = 96/120 trials, lower 95% limit 73%, P < 0.0001). There was no difference in error rates between the two formats (P = 0.53).ConclusionsThe use of touch-screen user interfaces in healthcare is increasingly common. Arrangement of drugs names in a categorical display format in the medication order-entry touch screen of an AIMS can result in faster data entry compared to an alphabetical arrangement of drugs. Results of this quality improvement project were used in our department’s design of our final intraoperative electronic anesthesia record. This testing approach using cognitive and usability engineering methods can be used to objectively design and evaluate many aspects of the clinician-computer interaction in electronic health records.

Work Action Analysis

Cognitive engineering methods must be extended to allow for analysis of educational systems. Specifically, this article proposes work action analysis (WAA) as a method for modeling educational systems to support their design and evaluation. WAA is defined as a form of work analysis that describes a system as the overall system goals, the collective work environments for each of multiple roles in the system, the essential interrelationships between the agents’ cognitive and physical activities and their environment, and the interactions between agents’ roles. A general description of WAA is given along with specific steps to generate a WAA model. WAA is used to model an undergraduate engineering course and guide its evaluation via a variety of measures. Finally, insights into educational systems from the WAA modeling method are discussed.

User Created Cognitive Artifacts: What Can They Teach Us About Design of Information Technology?

Cognitive artifacts are created and used to support task performance in many domains. These artifacts may be essential components designed into a process, they may have been created by users as work-arounds to system shortcomings, or they may be extensions to systems that add functionalities to meet evolving needs. Examination of cognitive artifacts may provide insights that complement other cognitive engineering methods such as task analysis. This panel will present findings from study of cognitive artifacts used in high risk domains such as healthcare. Panelists will address questions relating to research methodologies, theoretical frameworks, and design implications, such as: What are the roles of user-designed artifacts for care coordination and patient safety? What can designers learn from artifacts? How can we use artifact analysis to design better health information technologies? Can vendor-designed information tools and user-designed artifacts work in harmony to provide safe care? Panel members wi...

User Created Cognitive Artifacts: What can they Teach us about Design of Information Technology?

Cognitive artifacts are created and used to support task performance in many domains. These artifacts may be essential components designed into a process, they may have been created by users as work-arounds to system shortcomings, or they may be extensions to systems that add functionalities to meet evolving needs. Examination of cognitive artifacts may provide insights that complement other cognitive engineering methods such as task analysis. This panel will present findings from study of cognitive artifacts used in high risk domains such as healthcare. Panelists will address questions relating to research methodologies, theoretical frameworks, and design implications, such as: What are the roles of user-designed artifacts for care coordination and patient safety? What can designers learn from artifacts? How can we use artifact analysis to design better health information technologies? Can vendor-designed information tools and user-designed artifacts work in harmony to provide safe care? Panel members will address these questions based on their research studies and experiences within as well as outside the health care area.

Cognitive Engineering for Geographic Information Science

AbstractGeographic information science provides the foundation for the development of geospatial tools and services that support people in their spatio‐temporal decision‐making. In order to offer useful and useable solutions, principles of human spatial cognition regarding the representation and processing of spatial and temporal aspects of phenomena must be considered in the design of these tools. Such cognitively engineered geospatial services aim for cognitive adequacy and therefore facilitation of user interaction. This article argues for the necessity of cognitive engineering methods in the field of geographic information science by explicating their theoretical foundation and demonstrating practical geospatial applications. It further provides a framework for classifying cognitive user parameters, which can be employed for the personalization of geospatial services.

Applying a Warfighter-Centric System Design Process to a Department of Defense Acquisition Program

This case study describes a user-centered system design process that was undertaken by a team of multidisciplinary engineers supporting the Joint Tactical Radio System Handheld, Manpack and Small Form Fit Program. Motivated by the need to manage the risk associated with designing a compact, software-programmable radio to meet the diverse needs of the joint forces, the team applied cognitive engineering methods at key points during the system development and demonstration phase of the acquisition process. To make sound, data-driven design decisions, the team conducted four research efforts with warfighters to understand their tactical communication needs and the impact that proposed designs would have on mission and individual performance. Key insights and lessons learned from the application of cognitive engineering methods during the process and their impact on system design are discussed.

Cognitive Engineering in Radiation Screening for Homeland Security

Cognitive engineering methods in a large-scale homeland security program are described. Screening cargo containers at seaports for illicit radioactive material involves substantial staff, technology, and human operator decision making. The systems engineering model used by the radiation portal monitor (RPM) program entails elements similar to the incremental commitment model, which provides a risk-based focus for applications of cognitive engineering. This model includes methods for defining opportunities and context of use, methods for requirements and design solutions, and methods for evaluation. The application examples illustrate each of these general areas through work domain analysis, signal detection modeling, design of displays for radiation threat likelihood, and evaluation of the design concepts with field test data. The signal detection analysis shows how system detection sensitivity requirements for design can be established for low-probability, high-consequence events such as threats. The threat display concept developed by cognitive engineers has been implemented in new fielded systems. The success of cognitive engineering in the RPM program is largely a result of working with an incremental approach to systems engineering. This perspective provides application focus to those areas most likely to benefit from cognitive engineering within the overall constraints of the work domain, program schedule, and budget.