User's Cognitive Load Research Articles

Cognitive effort detection for tele-robotic surgery via personalized pupil response modeling.

Gaze tracking and pupillometry are established proxies for cognitive load, giving insights into a user's mental effort. In tele-robotic surgery, knowing a user's cognitive load can inspire novel human-machine interaction designs, fostering contextual surgical assistance systems and personalized training programs. While pupillometry-based methods for estimating cognitive effort have been proposed, their application in surgery is limited by the pupil's sensitivity to brightness changes, which can mask pupil's response to cognitive load. Thus, methods considering pupil and brightness conditions are essential for detecting cognitive effort in unconstrained scenarios. To contend with this challenge, we introduce a personalized pupil response model integrating pupil and brightness-based features. Discrepancies between predicted and measured pupil diameter indicate dilations due to non-brightness-related sources, i.e., cognitive effort. Combined with gaze entropy, it can detect cognitive load using a random forest classifier. To test our model, we perform a user study with the da Vinci Research Kit, where 17 users perform pick-and-place tasks in addition to auditory tasks known to generate cognitive effort responses. We compare our method to two baselines (BCPD and CPD), demonstrating favorable performance in varying brightness conditions. Our method achieves an average true positive rate of 0.78, outperforming the baselines (0.57 and 0.64). We present a personalized brightness-aware model for cognitive effort detection able to operate under unconstrained brightness conditions, comparing favorably to competing approaches, contributing to the advancement of cognitive effort detection in tele-robotic surgery. Future work will consider alternative learning strategies, handling the difficult positive-unlabeled scenario in user studies, where only some positive and no negative events are reliably known.

On validating a generic camera‐based blink detection system for cognitive load assessment

AbstractDetecting the human operator's cognitive state is paramount in settings wherein maintaining optimal workload is necessary for task performance. Blink rate is an established metric of cognitive load, with a higher blink frequency being observed under conditions of greater workload. Measuring blink rate requires the use of eye‐trackers which limits the adoption of this metric in the real‐world. The authors aim to investigate the effectiveness of using a generic camera‐based system as a way to assess the user's cognitive load during a computer task. Participants completed a mental task while sitting in front of a computer. Blink rate was recorded via both the generic camera‐based system and a scientific‐grade eye‐tracker for validation purposes. Cognitive load was also assessed through the performance in a single stimulus detection task. The blink rate recorded via the generic camera‐based approach did not differ from the one obtained through the eye‐tracker. No meaningful changes in blink rate were however observed with increasing cognitive load. Results show the generic‐camera based system may represent a more affordable, ubiquitous means for assessing cognitive workload during computer task. Future work should further investigate ways to increase its accuracy during the completion of more realistic tasks.

Strategies to reduce visual attention changes while learning and training in extended reality environments

Computer-Based training (CBT) is a widely used strategy in interactive learning and skills training to provide the trainee with information while in training. Extended Reality (XR) is a set of technologies used in the fourth industrial revolution for this purpose. However, an important XR limitation is the cognitive overload in trainees due to continuous changes in attention. The effects of the changes in attention on cognitive load have been studied in environments such as printed material and desktop PC’s. However, such effects are not yet fully known on XR platforms. This study’s aim was to identify strategies to reduce the changes in attention in instructional/educational materials, which use extended reality to present information to trainees. The presented information can guide the use of combined strategies to reduce the cognitive overload generated by XR display platforms. Therefore, an extensive literature review was carried out. From a set of 1946 articles, 53 studies were selected. The selected studies evaluated the cognitive load in instructional materials that used XR as a visualization platform. The results showed three strategy groups: those associated with the spatial integration of information, those associated with the use of visual features and those associated with the content segmentation. Effects of this strategies in the user performance and user cognitive load are discussed.

Effects of interface layout design on mobile learning efficiency: a comparison of interface layouts for mobile learning platform

PurposeThe purpose of this study is to explore the impact of mobile learning platforms on users' study efficiency and develop cognitive indicators to evaluate users' study efficiency on mobile learning platforms.Design/methodology/approachLayout style was the only independent factor that was investigated. A between-group experimental design was employed. Eye movement data were recorded during the experiment, following which participants were asked to complete an after-scenario questionnaire. This study evaluated the usability of the proposed new design using both subjective and objective data. The computer system usability questionnaire V3 (CSUQ) was used to measure subjective data. For the eye-tracking measure, gaze entropy, the proportion of fixation count and duration of each AOI were calculated. Gaze entropy reflects the complexity of information organization. Fixation counts and AOI duration represent the difficulty of information processing and attention distribution, respectively during the task.FindingsThe results indicated that interface layout presents significant effects on user's learning efficiency, usability and cognitive load. Sequential layout improved efficiency and satisfaction among participants and reduced information complexity. The results provided useful insights for designers whose goal is to improve user's learning efficiency under mobile learning scheme.Originality/valueThis study investigated the effects of interface layout on usability, user performance and cognitive load using subjective ratings and eye-tracking technology. Gaze entropy was used to measure the complexity of information organized by the interface design. Fixation count and duration proportion were used to identify the difficulty of information processing and distinguish users' distribution of cognitive resources. The results indicated that a vertical layout panel design was more efficient than a horizontal layout panel design. The design implications of the eye tracking indicators and research results were then summarized. This study is expected to encourage designers to optimize their design proposals using eye tracking testing.

Effects of Font Style and Font Color in News Text on User Cognitive Load in Intelligent User Interfaces

The advances on the Internet and news media technologies suggest intelligent and personalised media interfaces in order to improve reading efficiency of news readers and enhance news disseminations. Cognitive load is one of factors that affect the understanding ability of news readers and therefore news dissemination. It is expected that the display elements in an intelligent user interface of news media could be automatically adjusted to modulate the audience’s perceived cognitive load level in order to improve the news reading efficiency. While the font style and font colour of news text are the main display elements in the user interface of news media, it is not clear how these elements affect perceived cognitive load of news readers. This paper investigates perceived cognitive load of news readers under different font style and font colour conditions. Experiments with the news text in Chinese as a case study found that the change of text font style of keywords resulted in the increase of the reader’s perceived cognitive load during reading news text under the low introduced cognitive load and blue colour displayed keywords. While under the high introduced cognitive load, the italic font and red colour of keywords text resulted in the decrease of the reader’s perceived cognitive load significantly during reading news text, which therefore improves the news reading efficiency. This paper is limited to the effects of two aspects of news text (font styles and colours) on user’s perceived cognitive load. The future work will focus on the investigation of effects of other factors such as news pictures and their colours, numbers, and locations on user’s perceived cognitive load.

Designing Human-Centered 3D Gis Interface and Interaction Model to Support Agriculture Commodity Selection

The purpose of this study is to design an interface and an interaction model using user-centered Design approach. The model-built first aim is to assist the development process of a 3-dimensional geographic information system (3D GIS). The second aim is to ensure that the system that is built meets the needs of the user. GIS has been applied in various fields. In the agriculture field, GIS is often used to assist land evaluation and commodity selection. 3D GIS can provide map visualizations that are similar to actual earth topography. This advantage can minimize the user's cognitive load when using the map. Developing 3D GIS requires more complex components than 2D GIS. This study proposes an interface and interaction model for 3D GIS in scope of agricultural commodities selections. This study has five main stages. The first stage is software requirements Analisys. The second stage is data and content analysis. At this stage, data analysis is carried out to then build a data model to manage geocontent. The third stage is User Interface design. At this stage, an interface model is created as guidelines to arrange the 3D map layout. The fourth stage is the development of the interaction model to analysis of potential user actions and feedback of the system. The last stage is prototype development and evaluation. The prototype is evaluated by involving farmers to measure the adequacy of the data. A heuristic evaluation focused on eight statements was carried out. Based on the evaluation process, in general, most users agree with the statements made. It means that the model built meets the needs of the user. This model is expected to make it easier for software developers to build 3D GIS and ensure the farmer’s need in the sistems are meets. It’s hoped that combining visual elements and interaction can increase the usability level of 3D GIS

Information integration and instruction authoring of augmented assembly systems

Augmented assembly (AA) is being gradually incorporated into manual assembly owing to its abundant instruction forms, high efficiency of information transmission, and robust interactivity. However, a lack of a unified description model for assembly information and irregular instruction design forms leads to difficulties in instruction authoring and generation, inconveniences for data management, and low system portability. To address these issues, this paper presents an information-integration and instruction-authoring method for AA systems. First, design guidelines for the AA instructions are established with the aim of minimizing the user's cognitive load. After that, the designing of visual elements is performed based on the design guidelines, and the information model of the assembly information and instructions is built using Unified Modeling Language. Based on Extensible Markup Language files and Unity3D engine, a standard and rapid development process is presented for deploying the AA system to Hololens2 devices. The usability and replicability of the designed system are demonstrated, and a user study is carried out. The user's task performance is evaluated using completion time and errors as metrics. The NASA-task load index scale is used to estimate the user's workload, and the Likert scale reflects the user's subjective perception of the system and instructions. The results show that the proposed AA instructions can effectively improve the user's task performance; reduce their workload, especially the cognitive load; and improve their assembly experience.

Effects of auditory feedback and task difficulty on the cognitive load and virtual presence in a virtual reality dental simulation

This research examined the difference in cognitive load and the virtual presence depending on auditory feedback and task difficulty in haptic-based dental simulation. In the field of dental education, practice-centered training using handpiece has been crucial because a practitioner's psychomotor experience has a significant impact on the mastery of treatment skills. For the novice, it is necessary to reduce errors in dental treatment to enhancing skill acquisition in the haptic practice. In the training process, the force-feedback is crucial to elaborate subtle movement to guide what to do and how it should be hard or soft. However, It is not easy to add force-feedback to generate kinetic experience training. As an alternative method, we examined that auditory feedback can help learners' skill training. In this study, we analyzed how the presence/absence of auditory feedback at the different levels of task difficulty impacts learners' psychological demand and virtual presence in the virtual reality simulation. For this study, 29 dental college students participated in a dental simulation. The participants were grouped into two conditions that are with and without auditory feedback. Additionally, two consecutive tooth preparation tasks with different levels of difficulty were used in the simulation. The auditory feedback condition gives alarms to a learner when he treats a non-targeted tooth with a virtual handpiece. The user's cognitive load and virtual presence were measured to examine the effects of auditory feedback. The results revealed that the main effect was found in cognitive loads. Also, a significant interaction effect was shown in the virtual presence. We discussed the effective design methods for the virtual reality-based dental simulation through the result of this study.

A framework to estimate cognitive load using physiological data

Cognitive load has been widely studied to help understand human performance. It is desirable to monitor user cognitive load in applications such as automation, robotics, and aerospace to achieve operational safety and to improve user experience. This can allow efficient workload management and can help to avoid or to reduce human error. However, tracking cognitive load in real time with high accuracy remains a challenge. Hence, we propose a framework to detect cognitive load by non-intrusively measuring physiological data from the eyes and heart. We exemplify and evaluate the framework where participants engage in a task that induces different levels of cognitive load. The framework uses a set of classifiers to accurately predict low, medium and high levels of cognitive load. The classifiers achieve high predictive accuracy. In particular, Random Forest and Naive Bayes performed best with accuracies of 91.66% and 85.83% respectively. Furthermore, we found that, while mean pupil diameter change for both right and left eye were the most prominent features, blinking rate also made a moderately important contribution to this highly accurate prediction of low, medium and high cognitive load. The existing results on accuracy considerably outperform prior approaches and demonstrate the applicability of our framework to detect cognitive load.

Integrated displays to improve chronic disease management in ambulatory care: A SMART on FHIR application informed by mixed-methods user testing.

The study sought to evaluate a novel electronic health record (EHR) add-on application for chronic disease management that uses an integrated display to decrease user cognitive load, improve efficiency, and support clinical decision making. We designed a chronic disease management application using the technology framework known as SMART on FHIR (Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources). We used mixed methods to obtain user feedback on a prototype to support ambulatory providers managing chronic obstructive pulmonary disease. Each participant managed 2 patient scenarios using the regular EHR with and without access to our prototype in block-randomized order. The primary outcome was the percentage of expert-recommended ideal care tasks completed. Timing, keyboard and mouse use, and participant surveys were also collected. User experiences were captured using a retrospective think-aloud interview analyzed by concept coding. With our prototype, the 13 participants completed more recommended care (81% vs 48%; P < .001) and recommended tasks per minute (0.8 vs 0.6; P = .03) over longer sessions (7.0 minutes vs 5.4 minutes; P = .006). Keystrokes per task were lower with the prototype (6 vs 18; P < .001). Qualitative themes elicited included the desire for reliable presentation of information which matches participants' mental models of disease and for intuitive navigation in order to decrease cognitive load. Participants completed more recommended care by taking more time when using our prototype. Interviews identified a tension between using the inefficient but familiar EHR vs learning to use our novel prototype. Concept coding of user feedback generated actionable insights. Mixed methods can support the design and evaluation of SMART on FHIR EHR add-on applications by enhancing understanding of the user experience.

Identification of Potential Task Shedding Events Using Brain Activity Data

In Human–Machine Teaming environments, it is important to identify potential performance drops due to cognitive overload. If identified correctly, they can help improve the performance of the human–machine system by offloading some tasks to less cognitively overloaded users. This can help prevent user error that can result in critical failures. Also, it can improve productivity by keeping the human operators at an optimal performance state. This paper explores a new method for identifying user cognitive load by a three-class classification using brain activity data and by applying a convolutional neural network and long short-term memory model. The data collected from a set of cognitive benchmark experiments were used to train the model, which was then tested on two separate datasets consisting of more ecologically valid task environments. We experimented with various models built with different benchmark tasks to explore which benchmark tasks were better suited for the prediction of task shedding events in these compound tasks that are more representative of real-world scenarios. We also show that this method can be extended across-tasks and across-subject pools.

Experiment on Cognitive Load of Gesture Interaction Based on Semiotics

Objective: To explore the amount of information contained in gesture symbols and its relationship with cognitive load, and to provide theoretical guidance for gesture interaction design based on semiotics. Methods: Firstly, a user-defined method was combined with oral analysis to collect gestures and establish a corresponding semantic knowledge base. Based on the knowledge base, the icon algebra theory is used to redesign the gesture symbols. Then through experiments combined with the NASA-TLX scale to measure the user’s cognitive load, and through data analysis to explore the relationship between the amount of information contained in the gesture symbol and it. Conclusion: The amount of information contained in gesture symbols has a significant correlation with the user’s cognitive load and response time. As the amount of gesture symbol information increases, the user’s cognitive load and response time also increase accordingly. Based on this, a regression equation of user cognitive load and information amount of gesture symbols is established, which provides theoretical support for the design of gesture symbols based on semiotics.

Research on Optimization Method of VR Task Scenario Resources Driven by User Cognitive Needs

Research was performed in order to improve the efficiency of a user’s access to information and the interactive experience of task selection in a virtual reality (VR) system, reduce the level of a user’s cognitive load, and improve the efficiency of designers in building a VR system. On the basis of user behavior cognition-system resource mapping, a task scenario resource optimization method for VR system based on quality function deployment-convolution neural network (QFD-CNN) was proposed. Firstly, under the guidance of user behavior cognition, the characteristics of multi-channel information resources in a VR system were analyzed, and the correlation matrix of the VR system scenario resource characteristics was constructed based on the design criteria of human–computer interaction, cognition, and low-load demand. Secondly, analytic hierarchy process (AHP)-QFD combined with evaluation matrix is used to output the priority ranking of VR system resource characteristics. Then, the VR system task scenario cognitive load experiment is carried out on users, and the CNN input set and output set data are collected through the experiment, in order to build a CNN system and predict the user cognitive load and satisfaction in the human–computer interaction in the VR system. Finally, combined with the task information interface of a VR system in a smart city, the application research of the system resource feature optimization method under multi-channel cognition is carried out. The results show that the test coefficient CR value of the AHP-QFD model based on cognitive load is less than 0.1, and the MSE of CNN prediction model network is 0.004247, which proves the effectiveness of this model. According to the requirements of the same design task in a VR system, by comparing the scheme formed by the traditional design process with the scheme optimized by the method in this paper, the results show that the user has a lower cognitive load and better task operation experience when interacting with the latter scheme, so the optimization method studied in this paper can provide a reference for the system construction of virtual reality.

Research on Multimodal Perceptual Navigational Virtual and Real Fusion Intelligent Experiment Equipment and Algorithm

Virtual experiment is an important field of human-computer interaction. With more and more virtual laboratories emerging, we found that problems regarding virtual experiments are rising. Such problems can be listed as follows: First, human-computer interaction has lower efficiency during the process of virtual experiment, which means the computer cannot understand the user's intention thus leading to incorrect operation. Second, there are less detections for false behavior during experiments. Third, the virtual laboratory's sense of operation and realism is not strong. In order to solve the above problems, the multimodal sensing navigation virtual and real fusion laboratory (MSNVRFL) was designed and implemented in this paper. We design a new set of experimental equipment with the function of cognition and study a multimodal fusion model and algorithm for chemical experiments, which are both finally verified and applied in MSNVRFL. By using multimodal fusion perception algorithm, the user's true intentions can be understood and the human-computer interaction efficiency can be improved. By carrying out a virtual experiment with the mold of virtual and real fusion, problems like resources wasting and dangers happened during experiment can be avoided, user's sense of operation and realism can be improved. In addition, teaching navigation and wrong operation behavior reminders are provided for users. The experimental result shows that our method can improve the efficiency of human-computer interaction, reduce the user's cognitive load, strengthen the user's sense of reality and operation and stimulate students' interest in learning.

Users’ Perceptions of Smartphone Weather Applications’ Usability

The advancements of smartphone technology have led to a significant growth of smartphone weather applications (i.e. apps) on the market. Due to the high risk and time-critical situations associated with severe weather conditions (e.g. floods, tornados, and thunderstorms), designing weather apps to consider users’ needs is critical for improving product usability. This paper employs a user-centered design (UCD) approach to investigate a sample of end users’ perceptions regarding the usability of smartphone weather apps. Focus group responses analyzed with a thematic analysis approach revealed four main themes representing key usability principles and design specifications. The four themes were (1) user cognitive load, (2) effectiveness, (3) efficiency of use, and (4) user perceptions. Several recommendations were suggested based on the findings which might assist developers in designing highly interactive and usable apps leading to increased user satisfaction.

Research on the Quantitative Method of Cognitive Loading in a Virtual Reality System

Aimed at the problem of how to objectively obtain the threshold of a user’s cognitive load in a virtual reality interactive system, a method for user cognitive load quantification based on an eye movement experiment is proposed. Eye movement data were collected in the virtual reality interaction process by using an eye movement instrument. Taking the number of fixation points, the average fixation duration, the average saccade length, and the number of the first mouse clicking fixation points as the independent variables, and the number of backward-looking times and the value of user cognitive load as the dependent variables, a cognitive load evaluation model was established based on the probabilistic neural network. The model was validated by using eye movement data and subjective cognitive load data. The results show that the absolute error and relative mean square error were 6.52%–16.01% and 6.64%–23.21%, respectively. Therefore, the model is feasible.

Decomposition Strategies for Constructive Preference Elicitation

We tackle the problem of constructive preference elicitation, that is the problem of learning user preferences over verylarge decision problems, involving a combinatorial space of possible outcomes. In this setting, the suggested configuration is synthesized on-the-fly by solving a constrained optimization problem, while the preferences are learned iteratively by interacting with the user. Previous work has shown that Coactive Learning is a suitable method for learning userpreferences in constructive scenarios. In Coactive Learning the user provides feedback to the algorithm in the form of an improvement to a suggested configuration. When the problem involves many decision variables and constraints, this type of interaction poses a significant cognitive burden on the user. We propose a decomposition technique for large preference-based decision problems relying exclusively on inference and feedback over partial configurations. This has the clear advantage of drastically reducing the user cognitive load. Additionally, part-wise inference can be (up to exponentially) less computationally demanding than inference over full configurations. We discuss the theoretical implications of working with parts and present promising empirical results on one synthetic and two realistic constructive problems.

Augmented reality display based on user behavior

The development and commercialization of smart glasses in recent years have made the exploration of one's surroundings with mobile augmented reality (MAR) browsers anytime and anywhere more practical. However, users often suffer from issues such as cognitive overload and inconvenient interactions when operating MAR browsers on smart glasses owing to the constraints of the screen resolution and size. To overcome these problems, this paper presents a user-behavior-driven augmented content display approach called iDisplay. First, user behaviors were modeled while the smart glasses were used. A user behavior perception algorithm that infers the current state of the user by crosschecking his/her past behavior and feature data extracted from the built-in sensors of the smart glasses was then developed. Five augmented content display patterns corresponding to the modeled user's behavior states were designed accordingly. To verify that iDisplay can be based upon the perceived user states to adaptively manage the smart glasses display, a prototype system was built to conduct a series of experiments. The experimental results show that iDisplay can accurately infer user states and manage augmented content display accordingly. A user study also shows that iDisplay can successfully reduce the user's cognitive load and split attention when searching for specific point-of-interest information while moving. Furthermore, all subjects claimed that iDisplay causes less dizziness during the experiments than the native overview + detail augmented reality interface.

Cognitive Cost of Using Augmented Reality Displays.

This paper presents the results of two cognitive load studies comparing three augmented reality display technologies: spatial augmented reality, the optical see-through Microsoft HoloLens, and the video see-through Samsung Gear VR. In particular, the two experiments focused on isolating the cognitive load cost of receiving instructions for a button-pressing procedural task. The studies employed a self-assessment cognitive load methodology, as well as an additional dual-task cognitive load methodology. The results showed that spatial augmented reality led to increased performance and reduced cognitive load. Additionally, it was discovered that a limited field of view can introduce increased cognitive load requirements. The findings suggest that some of the inherent restrictions of head-mounted displays materialize as increased user cognitive load.

Using Semantics to Automatically Generate Speech Interfaces for Wearable Virtual and Augmented Reality Applications

This paper presents a framework for automatically generating speech-based interfaces for controlling virtual and augmented reality (AR) applications on wearable devices. Starting from a set of natural language descriptions of application functionalities and a catalog of general-purpose icons, annotated with possible implied meanings, the framework creates both vocabulary and grammar for the speech recognizer, as well as a graphic interface for the target application, where icons are expected to be capable of evoking available commands. To minimize user's cognitive load during interaction, a semantics-based optimization mechanism was used to find the best mapping between icons and functionalities and to expand the set of valid commands. The framework was evaluated by using it with see-through glasses for AR-based maintenance and repair operations. A set of experimental tests were designed to objectively and subjectively assess first-time user experience of the automatically generated interface in relation to that of a fully personalized interface. Moreover, intuitiveness of the automatically generated interface was studied by analyzing the results obtained through trained users on the same interface. Objective measurements (in terms of false positives, false negatives, task completion rate, and average number of attempts for activating functionalities) and subjective measurements (about system response accuracy, likeability, cognitive demand, annoyance, habitability, and speed) reveal that the results obtained by the first-time users and experienced users with the proposed framework's interface are very similar, and their performances are comparable with those of both the considered references.