Adaptive Interactive Systems Research Articles

A framework for visual-based adaptive object-robot interaction of a mobile service robot

The COVID-19 pandemic (hereinafter “the pandemic”) necessitated social distancing measures and limited physical contact, prompting the exploration of alternative methods for tasks like object delivery. Mobile service robots emerged as a potential solution, offering a bridge between humans and various tasks. While existing techniques have been introduced to enable robots to deliver objects in an end-to-end manner, they come with limitations. Grippers, for instance, can deliver only one object per round, cabinet robots require manual speed tuning to keep the object in place, and object holders lack generalizability. Inspired by the idea of human nature to use a tray to deliver the object, we developed the Visual-Based Adaptive Interaction System (hereinafter “VAIS”), a novel learning system, to improve service delivery using visual information and a fast neural learning mechanism. Within this system, the robot learns the optimal angular rotational and linear translational moving speeds to effectively transport objects placed on a tray without an extra holder. The robot validates these learnt movements by successfully completing multiple-object delivery tasks along designated routes. The results exhibit that the robot can utilize online learning after a few attempts to determine its proper moving speed and deliver different objects to a given location.

Design of syntactic adaptive interactive system based on human perception state estimation within scenario context

Design of syntactic adaptive interactive system based on human perception state estimation within scenario context

Towards pilot’s situation awareness enhancement: A framework of adaptive interaction system and its realization

Adaptive interaction system in flight control always aims to enhance the pilot’s situation awareness (SA) to achieve human-in-the-loop control. Most adaptive interaction systems are always activated according to the pilot’s current workload state. However, the pilot may already lose important information during a high workload, and thus the corresponding reaction of the adaptive interaction system would lag. Moreover, most adaptive interaction systems adopt the expert’s knowledge as a reference to generate information. Still, the tacit knowledge that reveals the information interrelationship is seldom studied, despite being the foundation of the interactive information display. To solve the above problems, we proposed an adaptive interaction system architecture with three subsystems. Firstly, we developed a workload level prediction subsystem, where physiological parameters are used to predict future workload levels, thus avoiding interaction system lag; Secondly, we developed a tacit expert knowledge mining subsystem to discover the interrelationship hidden in the expert’s perceived information, which will guide the interactive information interface. Thirdly, we developed a tips information inference subsystem to provide the lost SA information based on expert knowledge and the pilot’s online perceived information. The effectiveness of the proposed system is verified via a comparative experiment utilizing the control interface of a remotely piloted aircraft.

A User Study for the Evaluation of Adaptive Interaction Systems for Inclusive Industrial Workplaces

In recent years, production systems have become highly sophisticated and complex. As a result, while on the one hand, the least-skilled labor has been partially displaced by machines, high-skilled labor is more required to supervise and control advanced automation systems. In many cases, the complexity of machines implies an increased complexity of human–machine interfaces (HMIs), which are the main point of contact between the operator and the machine. To enable effective use of HMIs and to enable their usage by workers with different knowledge and capabilities, novel design approaches have been proposed. In particular, in this article, we consider the approach developed in the framework of the European research project INCLUSIVE, which aimed at designing industrial HMIs that adapt to the skills and capabilities of human operators. As a case study, we consider an adaptive interaction system for the woodworking industry and present an extensive evaluation carried out in real production environment with shopfloor workers. The effectiveness of the INCLUSIVE approach has been assessed with subjective and objective measurements and compared to that of interaction systems customarily used in industry. Results have shown that users appreciated the INCLUSIVE system and largely preferred it over the customary system. Moreover, with regard to objective performance-related measurements, they performed better when using the INCLUSIVE system since they received tailored guidance during the considered working tasks. Note to Practitioners— This article was motivated by the fact that advanced automation systems are often highly complex for human operators. To address this problem, we focus on the importance of designing the automation system around users and discuss an approach for adaptive automation, called INCLUSIVE. Its main feature is that it adapts the human–machine interface (HMI) according to operator’s skills, capabilities, and current mental fatigue. In this article, we provide an extensive evaluation of the INCLUSIVE system, considering a company producing woodworking machines as a use case. Assessment was carried out in the company shopfloor, considering real workers, and the INCLUSIVE system was compared with the customary HMI running on the company machines. The results of our study suggest that adapting the interaction to operator’s needs allows better working performance while letting workers more satisfied with the use of the system.

Towards tacit knowledge mining within context: Visual cognitive graph model and eye movement image interpretation

Visual attention is one of the most important brain cognitive functions, which filters the rich information of the outside world to ensure the efficient operation of limited cognitive resources. The underlying knowledge, i.e., tacit knowledge, hidden in the human attention allocation performances, is context-related and is hard to be expressed by experts, but it is essential for novice operator training and interaction system design. Traditional models of visual attention allocation and corresponding analysis methods seldomly involve task contextual information or present the tacit knowledge in an explicit and quantified way. Thus, it is challenging to pass on the expert’s tacit knowledge to the novice or utilize it to construct an interaction system by employing traditional methods. Therefore, this paper first proposes a new model called the visual cognitive graph model based on graph theory to model the visual attention allocation associated with the task context. Then, based on this graph model, utilize the data mining method to reveal attention patterns within context to quantitatively analyze the operator’s tacit knowledge during operation tasks. We introduced three physical quantities derived from graph theory to describe the tacit knowledge, which can be used directly to construct an interaction system or operator training. For example, discover the essential information within the task context, the relevant information affecting critical information, and the bridge information revealing the decision-making process. We tested the proposed method in the example of flight operation, the comparison results with the traditional eye movement graph model demonstrate that the proposed visual cognitive model can compromise the task context. The comparison results with the statistical analysis method demonstrate that our tacit knowledge mining method can reveal the underlying knowledge hidden in the visual information. Finally, we give practical applications in the examples of operator training guidance and adaptive interaction system. Our proposed method can explore more in-depth knowledge of visual information, such as the correlations of different obtained information and the way operator obtains information, most of which are even not noticed by operators themselves.

Toward a web-based multi-criteria decision support system for the layered evaluation of interactive adaptive systems

Toward a web-based multi-criteria decision support system for the layered evaluation of interactive adaptive systems

Ontologies in human–computer interaction: A systematic literature review

Human–Computer Interaction (HCI) is a multidisciplinary area that involves a diverse body of knowledge and a complex landscape of concepts, which can lead to semantic problems, hampering communication and knowledge transfer. Ontologies have been successfully used to solve semantics and knowledge-related problems in several domains. This paper presents a systematic literature review that investigated the use of ontologies in the HCI domain. The main goal was to find out how HCI ontologies have been used and developed. 35 ontologies were identified. As a result, we noticed that they cover different HCI aspects, such as user interface, interaction phenomenon, pervasive computing, user modeling / profile, HCI design, interaction experience and adaptive interactive system. Although there are overlaps, we did not identify reuse among the 35 analyzed ontologies. The ontologies have been used mainly to support knowledge representation and reasoning. Although ontologies have been used in HCI for more than 25 years, their use became more frequent in the last decade, when ontologies address a higher number of HCI aspects and are represented as both conceptual and computational models. Concerning how ontologies have been developed, we noticed that some good practices of ontology engineering have not been followed. Considering that the quality of an ontology directly influences the quality of the solution built based on it, we believe that there is an opportunity for HCI and ontology engineering professionals to get closer to build better and more effective ontologies, as well as ontology-based solutions.

Autonomous recommender system for reconnaissance tasks using a swarm of UAVs and asynchronous shepherding

Several artificial intelligence-based systems have been proposed to help human operators in safety-critical domains such as air-traffic control systems, health and farming. In scenarios where multiple autonomous systems are utilised, the cognitive capacity of humans hits its limits quickly, and sometimes the success of these systems is obstructed. Designing systems to sustain human performance over longer periods through balancing the load between human operators and autonomous systems can reduce errors and improve overall performance. Future human-autonomy systems need to be designed in a way that makes them scalable in terms of capabilities and safety, yet flexible enough to adapt to the cognitive workload of human operators. In this paper, we propose an adaptive human swarm interaction system for maintaining human operator performance in supervisory-control aerial-surveillance tasks. A bio-inspired guidance and control shepherding agent is proposed to support the human operator to guide multiple assets simultaneously using a single point of control. The proposed system employs an adjustable autonomous recommender system to provide the human operator with timely recommendations in real-time sittings. The proposed system is designed in a flexible way that allows it to adapt its rate of providing these recommendations in response to the needs of human operators. Simulation-based experiments are conducted to validate and evaluate the proposed recommender system. Experimental results demonstrate the effectiveness and efficiency of the proposed recommender system in minimising control errors while maintaining low computational costs that allow it to be used in a real-time applications.

The INCLUSIVE System: A General Framework for Adaptive Industrial Automation

While modern production systems are becoming increasingly technologically advanced, the presence of human operators remains fundamental in industrial workplaces. To complement and enhance the capabilities of human workers, approaches based on adaptive automation have been introduced. They consist of adapting the behavior of the system according to the user’s capabilities and effort. In this article, we present a general holistic framework for adaptive automation, called INCLUSIVE, that assists the operator during working tasks. The system consists of three modules. First, a thorough characterization of the operator’s constitutional and situational condition is provided; based on this, properly tailored adaptation is given, and if necessary, further training and support are provided. The framework has been implemented and tested considering three industrial use cases, selected as representative of a wide area of interest for the industry in Europe, in terms of both production requirements and involved operators. Tests have been carried out in real production environments, considering real production tasks carried out by 53 shop-floor workers. Results have shown that workers’ satisfaction when using the INCLUSIVE system and their performances was increased with respect to customary interaction systems currently used in industries. Moreover, the achieved results were used to formulate a set of recommendations for the design and implementation of an adaptive interaction system in relation to ensuring worker satisfaction and system usability in an industrial environment, as well as performance requirements. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article was motivated by the fact that, despite modern advanced automation, human operators are still central in the manufacturing process. However, technological progress often causes challenging interaction with complex industrial systems. The goal of this article is to introduce a complete framework for adaptive automation, with the ultimate goal of facilitating the interaction of human operators with complex industrial systems. The framework relies on three modules: measurement of human capabilities, the adaption of the interaction system, and additional teaching and support. The three modules are discussed at a high level, independently of the target application. Moreover, to facilitate their application in specific working contexts, examples are provided with respect to three different industrial applications. Results of tests carried out with shop-floor operators show that implementing the proposed framework allows better working performance and increases worker satisfaction with the use of automation.

ВИКОРИСТАННЯ АДАПТИВНО-ІНТЕРАКТИВНИХ СИСТЕМ У ПРОЦЕСІ НАВЧАННЯ ПЕРСОНАЛУ

The active development of adaptive-interactive systems, in particular virtual, augmented and mixed reality, in the field of education makes it appropriate to study the possibilities of using these systems directly to train personnel. The reduction of costs and increase of the availability of adaptive-interactive systems, including those that can be used in the training process, encourages their use in a growing number of companies around the world. The aim of the article is to study the usage of modern adaptive-interactive systems in the field of personnel training, such as virtual, augmented and mixed reality, to determine their advantages and disadvantages, as well as prospects for mass implementation of these systems in the practice of personnel training. Different types of virtual reality systems are distinguished, the delimitation of which lies in the plane of ways and modes of their interaction with the user. The research methodology is based on the semantic analysis of the interpretation of different types of adaptive-interactive systems of personnel training and analysis of their usage cases on existing enterprises. The expediency of using virtual and augmented reality technologies as methods of active learning in order to study various subjects is described and examples of personnel training systems using mixed reality are presented. The advantages and disadvantages of using these technologies in staff training are analyzed. Examples of application of adaptive-interactive systems at Ukrainian and foreign enterprises are given. The limited supply of adaptive-interactive systems of personnel training by the developers of such software and hardware indicates a lack of relevant specialists and the unwillingness of domestic enterprises to invest in these systems. At the same time, modern digital technologies such as virtual reality, augmented reality and mixed reality using artificial intelligence provide in practice high efficiency of the educational process in the enterprise by increasing concentration and attention of learners, easiness of information retrieval, providing practical skills, independence and simultaneous safety of employee training. In response, the directions of further researches in implementation of modern technologies of training of the personnel of the enterprises in Ukraine are offered.

Predictive multi‐view content buffering applied to interactive streaming system

This Letter discusses the benefits of introducing Machine Learning techniques in multi-view streaming applications. Widespread use of machine learning techniques has contributed to significant gains in numerous scientific and industry fields. Nonetheless, these have not yet been specifically applied to adaptive interactive multimedia streaming systems where, typically, the encoding bit rate is adapted based on resources availability, targeting the efficient use of network resources whilst offering the best possible user quality of experience (QoE). Intrinsic user data could be coupled with such existing quality adaptation mechanisms to derive better results, driven also by the preferences of the user. Head-tracking data, captured from camera feeds available at the user side, is an example of such data to which Recurrent Attention Models could be applied to accurately predict the focus of attention of users within videos frames. Information obtained from such models could be used to assist a preemptive buffering approach of specific viewing angles, contributing to the joint goal of maximising QoE. Based on these assumptions, a research line is presented, focusing on obtaining better QoE in an already existing multi-view streaming system.

The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart Environments.

Technologies, such as smart sensors, actuators, and other kinds of devices, are often installed in our environments (e.g., our Homes) and available to integrate our daily lives. Despite their installation being motivated by the pursuit of automation and increased efficiency, making these environments usable, acceptable and enjoyable in a sustainable, energy efficient way is not only a matter of automation. Tackling these goals is a complex task demanding the combination of different perspectives including building and urban Architecture, Ubiquitous Computing and Human-Computer Interaction (HCI) to provide occupants with the means to shape these environments to their needs. Interaction is of paramount relevance in the creation of adequate relations of users with their environments, but it cannot be seen independently from the ubiquitous sensing and computing or the environment’s architecture. In this regard, there are several challenges to HCI, particularly in how to integrate this multidisciplinary effort. Although there are several solutions to address some of these challenges, the complexity and dynamic nature of the smart environments and the diversity of technologies involved still present many challenges, particularly for its development. In general, the development is complex, and it is hard to create a dynamic environment providing versatile and adaptive forms of interaction. To participate in the multidisciplinary effort, the development of interaction must be supported by tools capable of facilitating co-design by multidisciplinary teams. In this article, we address the development of interaction for complex smart environments and propose the AM4I architecture and framework, a novel modular approach to design and develop adaptive multiplatform multilingual multi-device multimodal interactive systems. The potential of the framework is demonstrated by proof-of-concept applications in two different smart environment contexts, non-residential buildings and smart homes.

Мофологический синтез комплекса имитационных моделей с учетом адаптивного информационного взаимодействия оператора с объектом управления

The analysis of functioning a power system from the point of view of the efficiency of trouble-free operation and a human factor impact upon the safety of system functioning is carried out. The simulator of the morphological synthesis of a training-imitation complex is presented for modeling breakdown and uncommon working modes is presented. There is offered a structure of the adaptive information interaction system (hybrid intelligence) of an operator with the object of control.

An Interactive Adaptive Learning System Based on Agile Learner-Centered Design

Adaptive and interactive Learning concepts has apprehended the interest of educational actors and partners, especially in higher education. However, the implementation of those concepts has faced many challenges, particularly in Interactive Adaptive Learning Systems (IALS). The present paper aims to

EvalCHOICE: A Decision Support Approach for the Usability Evaluation of Interactive Adaptive Systems

Successful usability evaluation of interactive adaptive systems requires the use of appropriate methods. Different usability evaluation methods exist in the literature. These methods can be applied in association with the individual layers and with the whole adaptive system. In each situation, it is essential to apply the appropriate evaluation methods. In this paper, we propose a novel decision support approach called EvalCHOICE. It guides evaluators in the choice of appropriate usability evaluation methods for individual layers and the adaptive system as a whole. Our proposal is applied to determine the suitable evaluation methods for a given Web-based adaptive system in the field of transport.

A situation-based multi-agent architecture for handling misunderstandings in interactions

Abstract During interactions, system actors may face up misunderstandings when their local states contain inconsistent data about the same fact. Misunderstandings in interactions are likely to reduce interactivity performances (deviation or deadlock) or even affect overall system behavior. In this paper, we characterize misunderstandings in interactions between system actors (that may be human users or system agents) in interactive adaptive systems. To deal with such misunderstandings and ensure state consistency, we present an agent-based architecture and a scenario structuring approach. The system includes several agents devoted to scenario unfolding, plot adaptation and consistency management. Scenario structuring is based on the notion of a situation that is an elementary building block dividing the interactions between systems’ actors into contextual scenes. This pattern supports not only scenario execution but consistency management as well. In order to organize and control interactions, the situation contextualizes interactions and activity of the system’s actors. It also includes prevention and tolerance agent-based mechanisms to deal with the misunderstandings and their causes. We validate our consistency management mechanisms using Uppaal simulation and provide some experimental results to show the effectiveness of our approach on an online distance learning case study

Adaptive Micro-Channel Real-Time Interactive Systems of Nanchang Road Congestion Based on Improved Dijkstra Algorithm

The key technology research and development projects of national vehiclenetworking system needs, software products and components, developed withindependent intellectual property rights and core technologies based onimproved Dijkstra algorithm Nanchang road congestion adaptive system based onmicro-channel real-time interaction; main contents of transportation forNanchang adaptive significance of congestion vehicle networking applications,the status quo and research problems. GPS has immediate feedback andmicro-channel platform for mutual exchange of powerful features. Thuscountermeasures to enhance vehicle networking applications, Jiangxi Science andTechnology Park level. Developed based on the GPSNanchang traffic congestion adaptive micro-channel platform system, thesoftware product has low prices, professional, practical, easy installation,simple operation, with independent intellectual property rights, perfectlysuited for automotive units practical operation. Has a strong competitiveadvantage. The research results can be widely used in the province actual carservice units (including the use of e-commerce), a system price of Wu Yuan,automotive service products can be formed in Jiangxi Province, the developmentof new industries and generate significant economic benefits.

Towards Implicit User Modeling Based on Artificial Intelligence, Cognitive Styles and Web Interaction Data

A key challenge of adaptive interactive systems is to provide a positive user experience by extracting implicitly the users' unique characteristics through their interactions with the system, and dynamically adapting and personalizing the system's content presentation and functionality. Among the different dimensions of individual differences that could be considered, this work utilizes the cognitive styles of users as determinant factors for personalization. The overarching goal of this paper is to increase our understanding about the effect of cognitive styles of users on their navigation behavior and content representation preference. We propose a Web-based tool, utilizing Artificial Intelligence techniques, to implicitly capture and find any possible relations between the cognitive styles of users and their characteristics in navigation behavior and content representation preference by using their Web interaction data. The proposed tool has been evaluated with a user study revealing that cognitive styles of users have an effect on their navigation behavior and content representation preference. Research works like the reported one are useful for improving implicit and intelligent user modeling in engineering adaptive interactive systems.

Understanding How Adolescents with Autism Respond to Facial Expressions in Virtual Reality Environments

Autism Spectrum Disorders (ASD) are characterized by atypical patterns of behaviors and impairments in social communication. Among the fundamental social impairments in the ASD population are challenges in appropriately recognizing and responding to facial expressions. Traditional intervention approaches often require intensive support and well-trained therapists to address core deficits, with many with ASD having tremendous difficulty accessing such care due to lack of available trained therapists as well as intervention costs. As a result, emerging technology such as virtual reality (VR) has the potential to offer useful technology-enabled intervention systems. In this paper, an innovative VR-based facial emotional expression presentation system was developed that allows monitoring of eye gaze and physiological signals related to emotion identification to explore new efficient therapeutic paradigms. A usability study of this new system involving ten adolescents with ASD and ten typically developing adolescents as a control group was performed. The eye tracking and physiological data were analyzed to determine intragroup and intergroup variations of gaze and physiological patterns. Performance data, eye tracking indices and physiological features indicated that there were differences in the way adolescents with ASD process and recognize emotional faces compared to their typically developing peers. These results will be used in the future for an online adaptive VR-based multimodal social interaction system to improve emotion recognition abilities of individuals with ASD.

An adaptive utility interactive photovoltaic system based on a flexible switch matrix to optimize performance in real-time

Abstract Reconfigurable architectures are of great interest to system designers to improve the system’s operation and efficiency. In this paper, we propose an adaptive utility interactive photovoltaic (PV) system based on a novel Flexible Switch array Matrix topology. This proposed system maximizes the generated power in real-time in response to operational conditions such as shading, soiling, mismatches, and module failure among others. The proposed system is a compromise in the utilization of power conditioning equipment to maximize energy capture and system efficiency. Simulation results demonstrate an average 13% improvement in efficiency when compared with the central inverter topology performance. A prototype system has been designed and tested. The experimental results validate the proposed topology and its benefits for a wide range of applications.