Human Intention Research Articles

A Vision-Based Human Digital Twin Modeling Approach for Adaptive Human–Robot Collaboration

Abstract Human–robot collaboration (HRC) has been identified as a highly promising paradigm for human-centric smart manufacturing in the context of Industry 5.0. In order to enhance both human well-being and robotic flexibility within HRC, numerous research efforts have been dedicated to the exploration of human body perception, but many of these studies have focused only on specific facets of human recognition, lacking a holistic perspective of the human operator. A novel approach to addressing this challenge is the construction of a human digital twin (HDT), which serves as a centralized digital representation of various human data for seamless integration into the cyber-physical production system. By leveraging HDT, performance and efficiency optimization can be further achieved in an HRC system. However, the implementation of visual perception-based HDT remains underreported, particularly within the HRC realm. To this end, this study proposes an exemplary vision-based HDT model for highly dynamic HRC applications. The model mainly consists of a convolutional neural network that can simultaneously model the hierarchical human status including 3D human posture, action intention, and ergonomic risk. Then, on the basis of the constructed HDT, a robotic motion planning strategy is further introduced with the aim of adaptively optimizing the robotic motion trajectory. Further experiments and case studies are conducted in an HRC scenario to demonstrate the effectiveness of our approach.

Subject-Independent EEG Classification of Motor Imagery Based on Dual-Branch Feature Fusion.

A brain computer interface (BCI) system helps people with motor dysfunction interact with the external environment. With the advancement of technology, BCI systems have been applied in practice, but their practicability and usability are still greatly challenged. A large amount of calibration time is often required before BCI systems are used, which can consume the patient's energy and easily lead to anxiety. This paper proposes a novel motion-assisted method based on a novel dual-branch multiscale auto encoder network (MSAENet) to decode human brain motion imagery intentions, while introducing a central loss function to compensate for the shortcomings of traditional classifiers that only consider inter-class differences and ignore intra-class coupling. The effectiveness of the method is validated on three datasets, namely BCIIV2a, SMR-BCI and OpenBMI, to achieve zero calibration of the MI-BCI system. The results show that our proposed network displays good results on all three datasets. In the case of subject-independence, the MSAENet outperformed the other four comparison methods on the BCIIV2a and SMR-BCI datasets, while achieving F1_score values as high as 69.34% on the OpenBMI dataset. Our method maintains better classification accuracy with a small number of parameters and short prediction times, and the method achieves zero calibration of the MI-BCI system.

Migration or not? Impact of individual and urban factors on migrant workers' willingness of household registration transfer

PurposeThe purpose of this paper is to explore the factors that influence migrant workers' household registration transfer willingness at both individual and urban levels and to provide empirical evidence on adjusting the household registration system to accommodate economic development and migrant workers' imbalances.Design/methodology/approachThis paper adopts a hierarchical nonlinear model and examines individual and urban influencing factors of migrant workers' household registration transfer willingness, based on the data from China Migrants Dynamic Survey (CMDS) and the Urban Statistical Yearbooks.FindingsThis paper shows that: (1) multi-factors, such as age, education, marital status, household demographics, industry and migrant workers' contract coverage, have significant effects on migrant workers' household registration transfer willingness; (2) The urban public service equalization indicators, such as regional economic, educational resources, medical care and ecological quality, have significant effects on migrant workers' willingness to transfer household registration; (3) The heterogeneity of migrant workers' willingness to transfer household registration is significant in central, eastern and western China.Research limitations/implicationsThe authors provide a fresh perspective on population migration research in China and other countries worldwide based on the pull–push migration theory, which incorporates both individual and macro (urban) factors, enabling a comprehensive examination of the factors influencing household registration transfer willingness. This hierarchical ideology and approach (hierarchical nonlinear model) could be extended to investigate the influencing factors of various other human intentions and behaviors.Originality/valueMicro approaches (individual perspective) have dominated existing studies examining the factors influencing migrant workers' household registration transfer willingness. The authors combine individual and urban perspectives and adopt a more comprehensive hierarchical nonlinear model to extend the empirical evidence and provide theoretical explanations for the above issues.

The Advance of GPTs and Language Model in Cyber Security

Nature language processing (NLP), one of the most remarkable machine learning techniques currently available, is gaining traction with the public and has achieved great success in many applications. Many companies have developed language models, such as BERT, BART models from Google, and GPT (generative pre-trained transformer) series models from OpenAI. GPT is an unsupervised learning model that generates responses and uses unsupervised pre-training and supervised fine-tuning. GPT-2 is a multitask unsupervised learner that completes tasks using an unsupervised pre-trained model, including a zero-shot setting. GPT-3 extends the few-shot learning approach introduced in GPT, which does not require any gradient updates or fine-tuning for specific tasks. InstructGPT focuses on the alignment that could fit human intention by fine-tuning with human feedback. The outputs of InstructGPT significantly improved in truthfulness and were slightly less toxic than GPT-3, but bias and simple mistakes still existed. This paper aims to provide a detailed overview of the technical advancements utilized in GPT, GPT2, GPT3, and InstructGPT, explore the techniques in different models, and focus on the applications in the cybersecurity aspect. This paper compares the upgrade of GPT models and summarizes the SecureBERT model’s effects on cyber security.

Cooperative Game-Based Approximate Optimal Control of Modular Robot Manipulators for Human-Robot Collaboration.

Major challenges of controlling human-robot collaboration (HRC)-oriented modular robot manipulators (MRMs) include the estimation of human motion intention while cooperating with a robot and performance optimization. This article proposes a cooperative game-based approximate optimal control method of MRMs for HRC tasks. A harmonic drive compliance model-based human motion intention estimation method is developed using robot position measurements only, which forms the basis of the MRM dynamic model. Based on the cooperative differential game strategy, the optimal control problem of HRC-oriented MRM systems is transformed into a cooperative game problem of multiple subsystems. By taking advantage of the adaptive dynamic programming (ADP) algorithm, a joint cost function identifier is developed via the critic neural networks, which is implemented for solving the parametric Hamilton-Jacobi-Bellman (HJB) equation and Pareto optimal solutions. The trajectory tracking error under the HRC task of the closed-loop MRM system is proved to be ultimately uniformly bounded (UUB) by the Lyapunov theory. Finally, experiment results are presented, which reveal the advantage of the proposed method.

Intended Human Arm Movement Direction Prediction using Eye Tracking

ABSTRACT Collaborative robots are becoming increasingly popular in industries, providing flexibility and increased productivity for complex tasks. However, the robots are still not interactive enough since they cannot yet interpret humans and adapt to their behaviour, mainly due to limited sensory input. Prediction of human movement intentions could be one way to improve these robots. This paper presents a system that uses a recurrent neural network to predict the intended human arm movement direction, solely based on eye gaze, utilizing the notion of uncertainty to determine whether to trust a prediction or not. The network was trained with eye tracking data gathered using a virtual reality environment. The presented deep learning solution makes predictions on continuously incoming data and reaches an accuracy of 70.7%, for predictions with high certainty, and correctly classifies 67.89% of the movements at least once. The movements are, in 99% of the cases, correctly predicted the first time, before the hand reaches the target and more than 24% ahead of time in 75% of the cases. This means that a robot could receive warnings regarding in which direction an operator is likely to move and adjust its behaviour accordingly.

The Spaces and Places of the Tourism Encounter. On Re-Centring the Human in a More-Than/Non-Human World

This paper will revalue the phenomenological understandings of the tourism encounter, inspired by spatial theories of intentionality. With a growing body of theory delving into the relational realm and the ways in which the body and our actions are relationally enmeshed in networks of more-than/non-human entities, this paper seeks to recentre human intentionality as the core of the tourism encounter to better address its political nature and relevance. Whilst thereby critiquing some of the propositions of relational ontology, the paper is not about rejecting these, but augmenting them through a focus on the intention to care. Thereby, the paper will explore the ways in which the tourism encounter can be re-storied as one for making spaces and places of conviviality through people relating to each other and their surroundings with particular intent imbued with care. Valuing care and how it can be narrated helps to make space for a plurality of futures which can in turn break the deadlock of tourism being conceived either as mass/over- or alternative tourism. Both of these and more exist at the same time in the same place.

Fluid Landscapes and the Insular Imaginary in Roy Jacobsen’s Barrøy-Series

"Fluid Landscapes and the Insular Imaginary in Roy Jacobsen’s Barrøy-series. The paper aims to investigate, through the perspective of the recent new materialist turn in the humanities, the island imaginary in Northern Norway. The research will revolve around Roy Jacobsen’s Barrøy-serien, a series of four books which depict the islands of Northern Norway. The whole world of the islanders seems to be encircled and ultimately enclosed by water. The title of the first volume in the series (The Unseen) points directly toward the socio- and geopolitical invisibility of the inhabitants of the little island, who seem to be outside any social system, living their life off the grid. Islands have been often portrayed in literature as such isolated microcosms, objects of colonial gaze and desire, powerless and inert. However, in Barrøy-serien, Roy Jacobsen seems to break this pattern, by ascribing agency to nonhuman (and more-than-human) systems like the island or the ocean. The story of the protagonist Ingrid seems to slowly become the story of the Barrøy island itself, which is never just the backdrop of human action and intention, but a force that has the power to shape human destinies. I argue that these narratives allow nonhuman entities like the island and the ocean independence and autonomy, acknowledging their enmeshment in human life. Keywords: blue ecocriticism, island studies, nonhuman studies, new materialisms, landscape, ocean studies"

Tri-HGNN: Learning triple policies fused hierarchical graph neural networks for pedestrian trajectory prediction

In complex and dynamic urban traffic scenarios, the accurate trajectory prediction of surrounding pedestrians with interactive behaviors plays a vital role in the self-driving system. Intrinsic factors and extrinsic factors will inevitably influence the pedestrians trajectory. Intrinsic factors such as pedestrians diversified intentions bring rich and diverse multi-modal future possibilities. Besides, extrinsic factors affecting the future trajectory are accompanied by context semantics such as interactions among pedestrians. However, most of the existing methods discuss two problems (interaction and intention) separately. Considering both two factors impact the trajectory of pedestrians, a Triple Policies Fused Hierarchical Graph Neural Networks (Tri-HGNN) is proposed to model spatial and temporal interactions and intentions among the whole scene of pedestrians at each time step and predict the multiple future trajectories. Tri-HGNN contains three different policies: (i) Extrinsic-level policy is used to extract spatial nodes embedding from the interaction graph of pedestrian trajectories by using the Graph Attention Network. (ii) Intrinsic-level policy adopts the Graph Convolutional Network to infer the human intention for more accurate prediction. Moreover, human intention is influenced by the intrinsic interaction generated among pedestrians, so we fuse the interaction features to grasp the influence of the extrinsic interaction. (iii) Basic-level policy then integrates the heuristic information obtained from other two policies and concatenates it with historical trajectories to make multiple predictions through Temporal Convolution Network. Experimental results show that our model improves performance compared with state-of-the-art methods on the ETH/UCY and SDD benchmarks.

Improving performance of human action intent recognition: Analysis of gait recognition machine learning algorithms and optimal combination with inertial measurement units

Human action intent recognition has become increasingly dependent on computational accuracy, real-time responsiveness, and model lightness. Model selection, data filtering, and experimental design are three critical factors for the recognition of human intention in research. However, the performance of machine learning algorithms can vary depending on factors such as sensor location, the number of sensors used, channel selection, and dimensional combinations. Moreover, the collection of adequate and balanced data in such scenarios can be challenging. To address this issue, we present a comparative analysis of 12 commonly used machine learning algorithms for human action intention recognition. The synthetic minority oversampling technique is applied to fill in missing data. Traversing all possible combinations would require conducting 686 experiments, which is a daunting task in terms of both cost and efficiency. To tackle this challenge, we employ an orthogonal experiment design based on the Quasi-horizontal method. Our analysis indicates that lightGBM outperforms other algorithms in recognizing eight human daily activities. Furthermore, we conduct a polar difference and variance analysis based on a comprehensive balanced multi-metric orthogonal experiment for lightGBM using various sensor combinations and dimensions. The optimal combinations of different sensor numbers in terms of position, channel, and dimension are derived using this approach. Notably, our experimental design reduces the number of experiments required to only 49 times.

Unit middleware for implementation of human–machine interconnection intelligent ecology construction

General speech recognition models require large capacity and strong computing power. Based on small capacity and low computing power to realize speech analysis and semantic recognition is a research area with great challenges for constructing intelligent ecology of the Internet of Things. For this purpose, we set up the unit middleware for the implementation of human–machine interconnection, namely human–machine interaction based on phonetics and semantics control for constructing intelligent ecology of the Internet of Things. First, through calculation, theoretical derivation and verification we present a kind of novel deep hybrid intelligent algorithm, which has realized speech analysis and semantic recognition. Second, it is to establish unit middleware using the embedded chip as the core on the motherboard. Third, it is to develop the important auxiliary tools writer-burner and cross-compiler. Fourth, it is to prune procedures and system, download, burn and write the algorithms and codes into the unit middleware and cross-compile. Fifth, it is to expand the functions of the motherboard, provide more components and interfaces, for example including RFID(Radio Frequency Identification, RFID), ZigBee, Wi-Fi, GPRS(General Packet Radio Services, GPRS), RS-232 serial port, USB(Universal Serial Bus, USB) interfaces and so on. Sixth, we take advantage of algorithms, software and hardware to make machines "understand" human speech and "think" and "comprehend" human intentions so as to implement human–machine interconnection, which further structure the intelligent ecology of the Internet of Things. At last, the experimental results denote that the unit middleware have very good effect, fast recognition speed, high accuracy and good stability, consequently realizing the intelligent ecology construction of the Internet of Things.

Touch-and-Heal

Affective touch plays an important role in human-robot interaction. However, it is challenging for robots to perceive various natural human tactile gestures accurately, and feedback human intentions properly. In this paper, we propose a data-driven affective computing system based on a biomimetic quadruped robot with large-format, high-density flexible pressure sensors, which can mimic the natural tactile interaction between humans and pet dogs. We collect 208-minute videos from 26 participates and construct a dataset of 1212 human gestures-dog actions interaction sequences. The dataset is manually annotated with an 81-tactile-gesture vocabulary and a 44-corresponding-dog-reaction vocabulary, which are constructed through literature, questionnaire, and video observation. Then, we propose a deep learning algorithm pipeline with a gesture classification algorithm based on ResNet and an action prediction algorithm based on Transformer, which achieve the classification accuracy of 99.1% and the 1-gram BLEU score of 0.87 respectively. Finally, we conduct a field study to evaluate the emotion regulation effects through tactile affective interaction, and compare it with voice interaction. The results show that our system with tactile interaction plays a significant role in alleviating user anxiety, stimulating user excitement and improving the acceptability of robotic dogs.

An effective multimodal representation and fusion method for multimodal intent recognition

Intent recognition is a crucial task in natural language understanding. Current research mainly focuses on task-specific unimodal intent recognition. However, in real-world scenes, human intentions are complex and need to be judged by integrating information such as speech, tone, expression, and action. Therefore, this paper proposes an effective multimodal representation and fusion method (EMRFM) for intent recognition in real-world multimodal scenes. First, text, audio, and vision features are extracted based on pre-trained BERT, Wav2vec 2.0, and Faster R-CNN. Then, considering the complementarity and consistency among the modalities, the modality-shared and modality-specific encoders are constructed to learn shared and specific feature representations of the modalities. Finally, an adaptive multimodal fusion method based on an attention-based gated neural network is designed to eliminate noise features. Comprehensive experiments are conducted on the multimodal intent recognition MIntRec benchmark dataset. Our proposed model achieves higher accuracy, precision, recall, and F1-score than state-of-the-art multimodal learning methods. We also conduct multimodal sentiment recognition experiments on the CMU-MOSI dataset, and our model still outperforms state-of-the-art methods. In addition, the experiment demonstrates that the model’s multimodal representation well learned the modality’s shared and specific features. The multimodal fusion of the model achieves adaptive fusion and effectively reduces possible noise interference.

Analysing and predicting a country’s entrepreneurial activity using insights from entrepreneurs’ digital footprint

PurposeStarting from intention–behaviour models and building upon the growing evidence that aggregated internet search query data represent a good proxy of human interests and intentions. The purpose of this study is to demonstrate that the internet search traffic information related to the selected key terms associated with establishing new businesses, reflects well the dynamics of entrepreneurial activity in a country and can be used for predicting entrepreneurial activity at the national level.Design/methodology/approachTheoretical framework is based on intention–behaviour models and supported by the knowledge spillover theory of entrepreneurship. Monthly data on new business registration from 2018 to 2021 is derived from the open database of the Russian Federal Tax Service. Terms of internet search interest are identified through interviews with the recent founders of new businesses, whereas the internet search query statistics on the identified terms are obtained from Google Trends and Yandex Wordstat.FindingsThe results suggest that aggregated data about web searches related to opening a new business in a country is positively correlated with the dynamics of entrepreneurial activity in the country and, as such, may be useful for predicting the level of that activity.Practical implicationsThe results may serve as a starting point for a new approach to measure, monitor and predict entrepreneurial activities in a country and can help in better addressing policymaking issues related to entrepreneurship.Originality/valueTo the best of the authors’ knowledge, this study is original in its approach and results. Building on intention–behaviour models, this study outlines, to the best of the authors’ knowledge, the first usage of big data for analysing the intention–behaviour relationship in entrepreneurship. This study also contributes to the ongoing debate about the value of big data for entrepreneurship research by proposing and demonstrating the credibility of internet search query data as a novel source of quality data in analysing and predicting a country’s entrepreneurial activity.

Vision-Based Recognition of Human Motion Intent during Staircase Approaching.

Walking in real-world environments involves constant decision-making, e.g., when approaching a staircase, an individual decides whether to engage (climbing the stairs) or avoid. For the control of assistive robots (e.g., robotic lower-limb prostheses), recognizing such motion intent is an important but challenging task, primarily due to the lack of available information. This paper presents a novel vision-based method to recognize an individual's motion intent when approaching a staircase before the potential transition of motion mode (walking to stair climbing) occurs. Leveraging the egocentric images from a head-mounted camera, the authors trained a YOLOv5 object detection model to detect staircases. Subsequently, an AdaBoost and gradient boost (GB) classifier was developed to recognize the individual's intention of engaging or avoiding the upcoming stairway. This novel method has been demonstrated to provide reliable (97.69%) recognition at least 2 steps before the potential mode transition, which is expected to provide ample time for the controller mode transition in an assistive robot in real-world use.

Socially intelligent machines that learn from humans and help humans learn.

A hallmark of human intelligence is the ability to understand and influence other minds. Humans engage in inferential social learning (ISL)by using commonsense psychology to learn from others and help others learn. Recent advances in artificial intelligence (AI) are raising new questions about the feasibility of human-machine interactions that support such powerful modes of social learning. Here, we envision what it means to develop socially intelligent machines that can learn, teach, and communicate in ways that are characteristic of ISL. Rather than machines that simply predict human behaviours or recapitulate superficial aspects of human sociality (e.g. smiling, imitating), we should aim to build machines that can learn from human inputs and generate outputs for humans byproactively considering human values, intentions and beliefs. While such machines can inspire next-generation AI systems that learn more effectively from humans (as learners) and even help humans acquire new knowledge (as teachers), achieving these goals will also require scientific studies of its counterpart: how humans reason about machine minds and behaviours. We close by discussing the need for closer collaborations between the AI/ML and cognitive science communities to advance a science of both natural and artificial intelligence. This article is part of a discussion meeting issue 'Cognitive artificial intelligence'.

Robot teaching system based on hand-robot contact state detection and motion intention recognition

This paper presents a robot teaching system based on hand-robot contact state detection and human motion intent recognition. The system can detect the contact state of the hand-robot joint and extracts motion intention information from the human surface electromyography (sEMG) signals to control the robot's motion. First, a hand-robot contact state detection method is proposed based on the fusion of the virtual robot environment with the physical environment. With the use of a target detection algorithm, the position of the human hand in the color image of the physical environment can be identified and its pixel coordinates can be calculated. Meanwhile, the synthetic images of the virtual robot environment are combined with those of the physical robot scene to determine whether the human hand is in contact with the robot. Besides, a human motion intention recognition model based on deep learning is designed to recognize human motion intention with the input of sEMG signals. Moreover, a robot motion mode selection module is built to control the robot for single-axis motion, linear motion, or repositioning motion by combining the hand-robot contact state and human motion intention. The experimental results indicate that the proposed system can perform online robot teaching for the three motion modes.

Reciprocal Consistency Prediction Network for Multi-Step Human Trajectory Prediction

A reliable prediction of traffic participants’ trajectories is challenging for automated driving. We attempt to integrate human trajectory’s temporal and spatial reciprocal consistency into the prediction network structure. Moreover, humans are intention-driven agents, and modelling human intentions will continuously provide more accurate and detailed information for future trajectory prediction. In this paper, we propose a Reciprocal Consistency Prediction Network (RCPNet), a bi-directional prediction network consisting of the forward and the backward with a similar structure. The prior work of human trajectory prediction is mainly a unidirectional process, while RCPNet fully uses the past and future trajectory information at reverse temporal scales. By training the backward part to optimize the whole network parameters based on the reciprocal consistency, we can improve the prediction accuracy of the forward network. In particular, the framework incorporates a destination variational auto-encoder (DVAE) to estimate the target intention of humans and generate multi-modal trajectory prediction. In order to adapt to the offline prediction situation, we refer to the calculation method of correlation between two variables to present “Predictive Confidence” to clarify the reliability of offline predicted trajectories. Experimental results show that the RCPNet improves state-of-the-art performance on the SDD and ETH/UCY benchmarks.

Dual Transformer Based Prediction for Lane Change Intentions and Trajectories in Mixed Traffic Environment

In a mixed traffic environment of human and autonomous driving, it is crucial for an autonomous vehicle to predict the lane change intentions and trajectories of vehicles that pose a risk to it. However, due to the uncertainty of human intentions, accurately predicting lane change intentions and trajectories is a great challenge. Therefore, this paper aims to establish the connection between intentions and trajectories and propose a dual Transformer model for the target vehicle. The dual Transformer model contains a lane change intention prediction model and a trajectory prediction model. The lane change intention prediction model is able to extract social correlations in terms of vehicle states and outputs an intention probability vector. The trajectory prediction model fuses the intention probability vector, which enables it to obtain prior knowledge. For the intention prediction model, the accuracy can be improved by designing the multi-head attention. For the trajectory prediction model, the performance can be optimized by incorporating intention probability vectors and adding the LSTM. Verified on NGSIM and highD datasets, the experimental results show that this model has encouraging accuracy. Compared with the model without intention probability vectors, the impact of the model on NGSIM dataset and highD dataset in RMSE is improved by 57.27% and 58.70% respectively. Compared with two existed models, evaluation metrics of the intention prediction can be improved by 7.40-10.09% on NGSIM dataset and 2.17-2.69% on highD dataset within advanced prediction time 1s. This method provides the insights for designing advanced perceptual systems for autonomous vehicles.

Lower Limb Exoskeleton for Rehabilitation with Flexible Joints and Movement Routines Commanded by Electromyography and Baropodometry Sensors.

This paper presents the development of an instrumented exoskeleton with baropodometry, electromyography, and torque sensors. The six degrees of freedom (Dof) exoskeleton has a human intention detection system based on a classifier of electromyographic signals coming from four sensors placed in the muscles of the lower extremity together with baropodometric signals from four resistive load sensors placed at the front and rear parts of both feet. In addition, the exoskeleton is instrumented with four flexible actuators coupled with torque sensors. The main objective of the paper was the development of a lower limb therapy exoskeleton, articulated at hip and knees to allow the performance of three types of motion depending on the detected user's intention: sitting to standing, standing to sitting, and standing to walking. In addition, the paper presents the development of a dynamical model and the implementation of a feedback control in the exoskeleton.