Human Applications Research Articles

WISe: Wireless Intelligent Sensing for Human-Centric Applications

Contactless sensors based on electromagnet-wave (EM-wave) have great potential for human centric applications due to privacy, convenience and affordability advantages. Traditional researches usually focus on specific topics, such as hardware designs or software algorithms. However, a practical wireless sensing system covers multiple domains and technologies. Hence, a systematic framework of Wireless Intelligent Sensing (WISe) is proposed, for the first time, and a low-noise EM-wave system architecture is presented in this paper. The proposed framework and architecture make use of intelligent resources to minimize the noise in hardware and digital signal processing for various human-centric practical applications. WISe empowered healthcare applications and in-vehicle safety monitoring are shown to demonstrate the effectiveness of the proposed framework and architecture.

Bayesian logical neural networks for human-centered applications in medicine.

Background: Medicine is characterized by its inherent uncertainty, i.e., the difficulty of identifying and obtaining exact outcomes from available data. Electronic Health Records aim to improve the exactitude of health management, for instance using automatic data recording techniques or the integration of structured as well as unstructured data. However, this data is far from perfect and is usually noisy, implying that epistemic uncertainty is almost always present in all biomedical research fields. This impairs the correct use and interpretation of the data not only by health professionals but also in modeling techniques and AI models incorporated in professional recommender systems. Method: In this work, we report a novel modeling methodology combining structural explainable models, defined on Logic Neural Networks which replace conventional deep-learning methods with logical gates embedded in neural networks, and Bayesian Networks to model data uncertainties. This means, we do not account for the variability of the input data, but we train single models according to the data and deliver different Logic-Operator neural network models that could adapt to the input data, for instance, medical procedures (Therapy Keys depending on the inherent uncertainty of the observed data. Result: Thus, our model does not only aim to assist physicians in their decisions by providing accurate recommendations; it is above all a user-centered solution that informs the physician when a given recommendation, in this case, a therapy, is uncertain and must be carefully evaluated. As a result, the physician must be a professional who does not solely rely on automatic recommendations. This novel methodology was tested on a database for patients with heart insufficiency and can be the basis for future applications of recommender systems in medicine.

An Energy-Efficient Hybrid Clustering Technique (EEHCT) for IoT-Based Multilevel Heterogeneous Wireless Sensor Networks

Internet-of-Things (IoT)-based Heterogeneous Wireless Sensor Network (HWSN) has emerged as a prevalent technology that plays a significant role in developing various human-centric applications. Like in a wireless sensor network (WSN), energy is also the most crucial resource in IoT-based HWSN. The researchers have proposed many works to achieve energy-efficient network operations by minimizing energy usage. A vast proportion of these works emphasize using the clustering approach, which has proved its worth to a great extent. However, most schemes require the repeated formation of clusters incurring a significant amount of nodes’ energy in the clustering process. The protocol design of such schemes also varies with the changing levels of heterogeneity. In this work, a hybrid clustering scheme- An Energy-Efficient Hybrid Clustering Technique (EEHCT) has been proposed for IoT-based HWSN that minimizes the energy consumption in clusters’ formation and distributes the network load evenly irrespective of the heterogeneity level to prolong network lifetime. It appropriately utilizes dynamic and static clustering strategies to formulate the load-balanced clusters in the network. EEHCT establishes its supremacy over state-of-the-art schemes via an extensive set of simulations and experimentation in terms of multiple network performance metrics like stability, throughput, and network lifetime. Like, it achieves a gain up to 90.27% with respect to network lifetime over its peers in the standard operating conditions and under varying network configurations. In addition to quantitative analysis, a statistical analysis has also been provided to demonstrate the formation of energy-balanced clusters through the proposed scheme.

Detection of abnormal human behavior using deep learning

The complete human body or the various limb postures are involved in human action. These days,Abnormal Human Activity Recognition (Abnormal HAR) is highly well noticed and surveyed in manystudies. However, because of complicated difficulties such as sensor movement, positioning, and so on,as well as how individuals carry out their activities, it continues to be a difficult process. Identifyingparticular activities benefits human-centric applications such as postoperative trauma recovery, gesturedetection, exercise, fitness, and home care help. The HAR system has the ability to automate orsimplify most of the people’s everyday chores. HAR systems often use supervised or unsupervisedlearning as their foundation. Unsupervised systems operate according to a set of rules, whereassupervised systems need to be trained beforehand using specific datasets. This study conducts detailedliterature reviews on the development of various activity identification techniques currently being used.The three methods—wearable device-based, pose-based, and smartphone sensor—are examined in thisinquiry for identifying abnormal acts (AAD). The sensors in wearable devices collect data, whereas thegyroscopes and accelerometers in smartphones provide input to the sensors in wearable devices. Tocategorize activities, pose estimation uses a neural network. The Anomalous Action Detection Dataset(Ano-AAD) is created and improved using several methods. The study examines fresh datasets andinnovative models, including UCF-Crime. A new pattern in anomalous HAR systems has emerged,linking anomalous HAR tasks to computer vision applications including security, video surveillance,and home monitoring. In terms of issues and potential solutions, the survey looks at vision-based HAR

EgoFish3D: Egocentric 3D Pose Estimation From a Fisheye Camera via Self-Supervised Learning

Egocentric vision has gained increasing popularity recently, opening new avenues for human-centric applications. However, the use of the egocentric fisheye cameras allows wide angle coverage but image distortion is introduced along with strong human body self-occlusion imposing significant challenges in data processing and model reconstruction. Unlike previous work only leveraging synthetic data for model training, this paper presents a new real-world EgoCentric Human Pose (ECHP) dataset. To tackle the difficulty of collecting 3D ground truth using motion capture systems, we simultaneously collect images from a head-mounted egocentric fisheye camera as well as from two third-person-view cameras, circumventing the environmental restrictions. By using self-supervised learning under multi-view constraints, we propose a simple yet effective framework, namely EgoFish3D, for egocentric 3D pose estimation from a single image in different real-world scenarios. The proposed EgoFish3D incorporates three main modules. 1) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">The third-person-view module</i> takes two exocentric images as input and estimates the 3D pose represented in the third-person camera frame; 2) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">the egocentric module</i> predicts the 3D pose in the egocentric camera frame; and 3) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">the interactive module</i> estimates the rotation matrix between the third-person and the egocentric views. Experimental results on our ECHP dataset and existing benchmark datasets demonstrate the effectiveness of the proposed EgoFish3D, which can achieve superior performance to existing methods.

Disjunctive Threshold Networks for Tabular Data Classification

While neural networks have been achieving increasingly significant excitement in solving classification tasks such as natural language processing, their lack of interpretability becomes a great challenge for neural networks to be deployed in certain high-stakes human-centered applications. To address this issue, we propose a new approach for generating interpretable predictions by inferring a simple three-layer neural network with threshold activations, so that it can benefit from effective neural network training algorithms and at the same time, produce human-understandable explanations for the results. In particular, the hidden layer neurons in the proposed model are trained with floating point weights and binary output activations. The output neuron is also trainable as a threshold logic function that implements a disjunctive operation, forming the logical-OR of the first-level threshold logic functions. This neural network can be trained using state-of-the-art training methods to achieve high prediction accuracy. An important feature of the proposed architecture is that only a simple greedy algorithm is required to provide an explanation with the prediction that is human-understandable. In comparison with other explainable decision models, our proposed approach achieves more accurate predictions on a broad set of tabular data classification datasets.

A study of bias mitigation strategies for speaker recognition

Speaker recognition is increasingly used in several everyday applications including smart speakers, customer care centers and other speech-driven analytics. It is crucial to accurately evaluate and mitigate biases present in machine learning (ML) based speech technologies, such as speaker recognition, to ensure their inclusive adoption. ML fairness studies with respect to various demographic factors in modern speaker recognition systems are lagging compared to other human-centered applications such as face recognition. Existing studies on fairness in speaker recognition systems are largely limited to evaluating biases at specific operating points of the systems, which can lead to false expectations of fairness. Moreover, there are only a handful of bias mitigation strategies developed for speaker recognition systems. In this paper, we systematically evaluate the biases present in speaker recognition systems with respect to gender across a range of system operating points. We also propose adversarial and multi-task learning techniques to improve the fairness of these systems. We show through quantitative and qualitative evaluations that the proposed methods improve the fairness of ASV systems over baseline methods trained using data balancing techniques. We also present a fairness-utility trade-off analysis to jointly examine fairness and the overall system performance. We show that although systems trained using adversarial techniques improve fairness, they are prone to reduced utility. On the other hand, multi-task methods can improve the fairness while retaining the utility. These findings can inform the choice of bias mitigation strategies in the field of speaker recognition.

Removing AI’s sentiment manipulation of personalized news delivery

Artificial intelligence (AI) is empowering personalized online news delivery to accommodate people’s information needs and combat information overload. However, AI models learned from user data are inheriting and amplifying some underlying human prejudice such as the sentiment bias of news reading, which may lead to potential negative societal effects and ethical concerns. Here, substantial evidence shows that AI is manipulating the sentiment orientation of news displayed to users by promoting the presence chance of negative news, even if there is no human interference. To mitigate this manipulation, a sentiment-debiasing method based on a decomposed adversarial learning framework is proposed, which can reduce 97.3% of sentiment bias with only 2.9% accuracy sacrifice. Our work provides the potential in improving AI’s responsibility in many human-centered applications such as online journalism and information spread.

An Audio-Seismic Fusion Framework for Human Activity Recognition in an Outdoor Environment

Human activity recognition has a significant impact on people’s daily lives. The need to infer human activities is prominent in many human-centric applications, such as healthcare and individual assistance. In this article, we introduce a noninvasive human activity recognition system that utilizes footstep-induced vibration and sound in an outdoor environment with the aim of achieving improved performance over a single source of information. We employ 1-D convolutional neural networks (1-D CNNs) for automated feature extraction, fusion, and activity recognition on a nine-class classification problem. The proposed framework reports an average F1 score of 92%, which corresponds to a 5.74% improvement over the best-performing state-of-the-art. Confusion matrix-based analysis demonstrates that audio-seismic fusion not only reduces misclassifications, but also reduces the impact of background noise on model performance. In addition, we demonstrate that a model trained on a balanced dataset has a higher F1 score than one trained on an imbalanced dataset. Activity-wise performance is reported to show the efficacy of the proposed fusion-based framework. We also contribute an audio-seismic dataset for human activity recognition in an outdoor environment. The dataset is collected in a variety of challenging environments, such as varying grass length, soil moisture content, and the passing of unwanted vehicles.

The concept of collaborative engineering: a systematic literature review

ABSTRACT Collaborative engineering is not a new subject but it assumes a new importance in the Industry 4.0 (I4.0). There are other concepts frequently mismatched with collaboration. Thus, the main objective of this paper is to put forward a collaborative engineering concept, along its sub concepts, supported by an extensive systematic literature review. A critical analysis and discussion about the fundamental importance of learning, and the central human role in collaboration, in the I4.0, is presented, based on the main insights brought through the literature review. This study also enables to realize about the importance of collaboration in the current digitalization era, along with the importance of recent approaches and technology for enabling or promoting collaboration. Main current practices of human centered and autonomous machine-machine approaches and applications of collaboration in engineering, namely in manufacturing and management, are presented, along with main difficulties and further open research opportunities on collaboration..

MultiCNN-FilterLSTM: Resource-efficient sensor-based human activity recognition in IoT applications

With the recent advances in the Internet of Things (IoT) technologies, various human-centered applications have proliferated and improved the quality of users’ life. In the meantime, human activity recognition (HAR) has been considered as an essential component of human-centered applications in IoT due to its capability of providing substantial information about the user states. Whereas deep learning-based HAR methods have been recently proposed, there are still rooms to improve the HAR models, particularly from the perspective of IoT, as the models need to be precise but resource efficient. In this paper, to support IoT systems that require a resource-efficient model, we thereby propose a deep learning-based HAR model, called MultiCNN-FilterLSTM that combines a multihead convolutional neural network (CNN) with a long–short-term memory (LSTM) through a residual connection in which feature vectors are efficiently processed in hierarchical order. Accordingly, a novel approach to using LSTM cells, which we call filterwise LSTM (FilterLSTM), is proposed whereby the HAR model can learn the dependencies among the features at different hierarchical levels. The proposed HAR model has been exhaustively evaluated on two publicly available datasets. The proposed HAR model enhances the classification accuracy by 2.3%–4.4%, while it requires 21%–70% fewer operations than the state-of-the-art models. In addition, the proposed model is deployed to a Raspberry Pi 4 for further analysis in terms of deployment. The experimental results are presented to verify the merits of the proposed HAR model compared to the state-of-art models by exploiting hierarchical relationships and to deliver significant insights on the effectiveness of the proposed FilterLSTM from the perspective of the IoT system.

Multiple Thermal Sensor Array Fusion Toward Enabling Privacy-Preserving Human Monitoring Applications

Human-centric applications of a single thermal sensor array (TSA) have performed extremely well in many areas. However, most of these works have not yet reached the real applicability stage of the Internet of Things (IoT) applications. The main limitation of deploying such systems on a large scale is the challenge of fusing multiple TSAs to cover a wide inspection area, e.g., smart homes, hospitals, and many other domestic environments. On the other hand, objects that appear in the low-resolution thermal images acquired from TSA have low intraclass variations and high interclass similarities, making the identification of the overlapping regions through matching a comparable template image in multiple images very difficult. This article proposes a motion-based approach to fuse multiple TSAs and learn the domestic environment layout to enable further human-centred IoT applications to run in the cloud. Besides, a privacy improvement on utilizing these sensors in IoT applications is proposed. The proposed approach is evaluated with comprehensive experiments on different sensor placements and domestic environment conditions. This article shows an average performance of 92.5% accuracy using various machine learning techniques and use case scenarios.

A Conditional GAN for Generating Time Series Data for Stress Detection in Wearable Physiological Sensor Data.

Human-centered applications using wearable sensors in combination with machine learning have received a great deal of attention in the last couple of years. At the same time, wearable sensors have also evolved and are now able to accurately measure physiological signals and are, therefore, suitable for detecting body reactions to stress. The field of machine learning, or more precisely, deep learning, has been able to produce outstanding results. However, in order to produce these good results, large amounts of labeled data are needed, which, in the context of physiological data related to stress detection, are a great challenge to collect, as they usually require costly experiments or expert knowledge. This usually results in an imbalanced and small dataset, which makes it difficult to train a deep learning algorithm. In recent studies, this problem is tackled with data augmentation via a Generative Adversarial Network (GAN). Conditional GANs (cGAN) are particularly suitable for this as they provide the opportunity to feed auxiliary information such as a class label into the training process to generate labeled data. However, it has been found that during the training process of GANs, different problems usually occur, such as mode collapse or vanishing gradients. To tackle the problems mentioned above, we propose a Long Short-Term Memory (LSTM) network, combined with a Fully Convolutional Network (FCN) cGAN architecture, with an additional diversity term to generate synthetic physiological data, which are used to augment the training dataset to improve the performance of a binary classifier for stress detection. We evaluated the methodology on our collected physiological measurement dataset, and we were able to show that using the method, the performance of an LSTM and an FCN classifier could be improved. Further, we showed that the generated data could not be distinguished from the real data any longer.

Human-in-the-loop virtual reality offloading scheme in wireless 6G Terahertz networks

Upcoming sixth-generation networks have to be carefully designed to offer support to a wide range of human-centric applications, typically computationally intensive and heavily dependent on human perception. This paper proposes a human-in-the-loop-based resource allocation approach found on the joint use of matching theory and stochastic network calculus principles to efficiently address tasks offloading issues within a sixth-generation network environment. The considered matching game approach is characterized by externalities, whose stability is properly discussed here. In particular, by focusing on virtual reality services, the paper aims at maximizing the average service reliability in terms of ability in guaranteeing a proper quality of experience target, i.e., keeping the end-to-end delay below a suitable threshold, by the proper exploitation of the human cognitive limitations. Finally, the effectiveness of the proposed human-in-the-loop-based resource allocation approach is verified by proving comparisons with simulation results and conventional brain-agnostic strategies.

Body-Powered and Portable Soft Hydraulic Actuators as Prosthetic Hands

Soft robotic actuators are highly flexible, compliant, dexterous, and lightweight alternatives that can potentially replace conventional rigid actuators in various human-centric applications. This research aims to develop a soft robotic actuator that leverages body movements to mimic the function of human fingers for gripping and grasping tasks. Unlike the predominantly used chamber-based actuation, this study utilizes actuators made from elastomers embedded with fiber braiding. The Young’s modulus of the elastomer and braiding angles of the fiber highly influenced the bending angle and force generated by these actuators. In this experiment, the bending and force profiles of these actuators were characterized by varying the combinations of elastomeric materials and braiding angles to suit hand manipulation tasks. Additionally, we found that utilizing water, which is relatively more incompressible than air, as the actuation fluid enabled easier actuation of the actuators using body movements. Lastly, we demonstrated a body-powered actuator setup that can provide comfort to patients in terms of portability, standalone capability, and cost effectiveness, potentially allowing them to be used in a wide range of wearable robotic applications.

Combating False Data Injection Attacks on Human-Centric Sensing Applications

The recent prevalence of machine learning-based techniques and smart device embedded sensors has enabled widespread human-centric sensing applications. However, these applications are vulnerable to false data injection attacks (FDIA) that alter a portion of the victim's sensory signal with forged data comprising a targeted trait. Such a mixture of forged and valid signals successfully deceives the continuous authentication system (CAS) to accept it as an authentic signal. Simultaneously, introducing a targeted trait in the signal misleads human-centric applications to generate specific targeted inference; that may cause adverse outcomes. This paper evaluates the FDIA's deception efficacy on sensor-based authentication and human-centric sensing applications simultaneously using two modalities - accelerometer, blood volume pulse signals. We identify variations of the FDIA such as different forged signal ratios, smoothed and non-smoothed attack samples. Notably, we present a novel attack detection framework named Siamese-MIL that leverages the Siamese neural networks' generalizable discriminative capability and multiple instance learning paradigms through a unique sensor data representation. Our exhaustive evaluation demonstrates Siamese-MIL's real-time execution capability and high efficacy in different attack variations, sensors, and applications.

Hierarchical Multi-Supervision Multi-Interaction Graph Attention Network for Multi-Camera Pedestrian Trajectory Prediction

Pedestrian trajectory prediction has become an essential underpinning in various human-centric applications including but not limited to autonomous vehicles, intelligent surveillance system and social robotics. Previous research endeavors mainly focus on single camera trajectory prediction (SCTP), while the problem of multi-camera trajectory prediction (MCTP) is often overly simplified into predicting presence in the next camera. This paper addresses MCTP from a more realistic yet challenging perspective, by redefining the task as a joint estimation of both future destination and possible trajectory. As such, two major efforts are devoted to facilitating related research and advancing modeling techniques. Firstly, we establish a comprehensive multi-camera Scenes Pedestrian Trajectory Dataset (mcScenes), which is collected from a real-world multi-camera space combined with thorough human interaction annotations and carefully designed evaluation metrics. Secondly, we propose a novel joint prediction framework, namely HM3GAT, for the MCTP task by building a tailored network architecture. The core idea behind HM3GAT is a fusion of topological and trajectory information that are mutually beneficial to the prediction of each task, achieved by deeply customized networks. The proposed framework is comprehensively evaluated on the mcScenes dataset with multiple ablation experiments. Status-of-the-art SCTP models are adopted as baselines to further validate the advantages of our method in terms of both information fusion and technical improvement. The mcScenes dataset, the HM3GAT, and alternative models are made publicly available for interested readers.

Editorial Special Issue Interaction With Artificial Intelligence Systems: New Human-Centered Perspectives and Challenges

The papers in this special section focus on the interaction with artificial intelligence (AI) systems using human-centered applications. AI methods are being applied to numerous areas, including medicine, security, transportation, industry, smart homes and cities, business, social sciences, and psychology. AI is currently a part of our daily lives. People interact continuously with AI: it is inside houses, computers, mobile phones, and applications. AI can make predictions and give suggestions for movies, songs, or future purchases based on our previous choices. It affects the society and economy. People are fascinated by AI in the ways it improves and facilitates human life (improving health care and discharging workers from heavy or dangerous jobs). People are also concerned with AI’s implementation risks, such as ethical, security, and privacy issues. There are also concerns that AI machines may replace humans in various activities. AI researchers and practitioners have been facing these issues and further research is needed to design technical and regulatory applicable solutions. This special issue (SI) investigates a broad range of issues deriving from human interaction with AI. We encouraged interdisciplinary and multidisciplinary contributions toward understanding how AI could improve human life in various fields.

Marfusion: An Attention-Based Multimodal Fusion Model for Human Activity Recognition in Real-World Scenarios

Human Activity Recognition(HAR) plays an important role in the field of ubiquitous computing, which can benefit various human-centric applications such as smart homes, health monitoring, and aging systems. Human Activity Recognition mainly leverages smartphones and wearable devices to collect sensory signals labeled with activity annotations and train machine learning models to recognize individuals’ activity automatically. In order to deploy the Human Activity Recognition model in real-world scenarios, however, there are two major barriers. Firstly, sensor data and activity labels are traditionally collected using special experimental equipment in a controlled environment, which means fitting models trained with these datasets may result in poor generalization to real-life scenarios. Secondly, existing studies focus on single or a few modalities of sensor readings, which neglect useful information and its relations existing in multimodal sensor data. To tackle these issues, we propose a novel activity recognition model for multimodal sensory data fusion: Marfusion, and an experimental data collection platform for HAR tasks in real-world scenarios: MarSense. Specifically, Marfusion extensively uses a convolution structure to extract sensory features for each modality of the smartphone sensor and then fuse the multimodal features using the attention mechanism. MarSense can automatically collect a large amount of smartphone sensor data via smartphones among multiple users in their natural-used conditions and environment. To evaluate our proposed platform and model, we conduct a data collection experiment in real-life among university students and then compare our Marfusion model with several other state-of-the-art models on the collected datasets. Experimental Results do not only indicate that the proposed platform collected Human Activity Recognition data in the real-world scenario successfully, but also verify the advantages of the Marfusion model compared to existing models in Human Activity Recognition.

Society 5.0: Human Centric, Decentralized, and Hyperautomated

The articles in this special section focus on the concept of Society 5.0. The five articles provide a glimpse of the human-centric applications and approaches in our continuously evolving digital society. With the rapid developments and proliferation of IoT, cyber-physical systems, AI, autonomous and mobility technologies, cyber (virtual) space and physical (real) space are converging. Such advancements in information and communication technology (ICT) have brought progress in industrial automation and economic growth. As things, people, and devices are increasingly connected, another digital revolution that focuses on human, social, and environmental needs is underway. Having its origin in Japan, Society 5.0 attempts to bring social and environmental concerns into the midst of a digital society. Conceptually, Society 5.0 is defined as “a human-centered (smart) society that balances economic advancement with the resolution of social problems by a system that highly integrates cyberspace and physical space.” a For a human-centered society, technologies can contribute significant value to many fields, such as mobility, healthcare and caregiving, manufacturing and infrastructures, food and agriculture, energy and climate, and disaster prevention. All the previous are in complete accordance with IEEE’s core purpose to foster technological innovation and excellence for the benefit of humanity.