Human-machine Communication Systems Research Articles

A Survey on Speech Emotion Recognition System using CNN Algorithm

The speech emotion recognition system (SER) plays an important role in decoding and predicting the speaker's emotional state by analyzing audio signals. Emotions are often simplified and grouped into categories such as anger, happiness, sadness, and even neutral emotional states. These emotions serve to communicate one's thoughts and provide insight into one's physical and mental health. Speech is the most basic and natural form of communication when interacting with others. Voice signals enable rapid communication between individuals, making them a valuable and effective method of expression. Over the past decades, countless research initiatives have been devoted to the development of voice-based automatic emotion recognition systems, especially to improve human-machine communication. Voice is gradually taking center stage in the field of Human-Machine interfaces in the IT sector. This interdisciplinary field draws on a variety of fields, including computer science, signal processing, psychology, linguistics, and more. As technology advances, it allows for seamless communication between humans and machines. Speech recognition not only interprets speech but also captures nuances in a person's tone and expressions, similar to body language. Therefore, it becomes an essential element of the Human-Machine communication system.

Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network

Sign Language Recognition is a breakthrough for communication among deaf-mute society and has been a critical research topic for years. Although some of the previous studies have successfully recognized sign language, it requires many costly instruments including sensors, devices, and high-end processing power. However, such drawbacks can be easily overcome by employing artificial intelligence-based techniques. Since, in this modern era of advanced mobile technology, using a camera to take video or images is much easier, this study demonstrates a cost-effective technique to detect American Sign Language (ASL) using an image dataset. Here, “Finger Spelling, A” dataset has been used, with 24 letters (except j and z as they contain motion). The main reason for using this dataset is that these images have a complex background with different environments and scene colors. Two layers of image processing have been used: in the first layer, images are processed as a whole for training, and in the second layer, the hand landmarks are extracted. A multi-headed convolutional neural network (CNN) model has been proposed and tested with 30% of the dataset to train these two layers. To avoid the overfitting problem, data augmentation and dynamic learning rate reduction have been used. With the proposed model, 98.981% test accuracy has been achieved. It is expected that this study may help to develop an efficient human–machine communication system for a deaf-mute society.

Implementation of Voice based Touchless Lift System

In this rapid world of technology where voice begins its era of domination to replace the touch screens from smart phones to huge computer systems, bringing voice in day to day affairs becomes significant. An elevator or lift is a transport vehicle that moves people or goods from on floor to another floor in a building. Typically push buttons were used to send requests to the elevators. In recent times touch buttons are coming to use. But now voice recognition can replace the push/touch technology. Elevators being one such system used in daily life serves this purpose of making future generations hands free which also becomes a boon for the disabled as well as helps during the pandemic situation to avoid physical contact. The main objective of this project is to propose and assemble a voice operated lift/elevator control system. The proposed system acts as human-machine communication system. This research combines electronic control technology with speech recognition technology. The input to the system is human speech. Speech recognition is the method of recognizing the vocal words to take the essential actions accordingly. This device is very helpful for paralysis, short height people and physically challenged persons.

Mechanoreceptor‐Inspired Dynamic Mechanical Stimuli Perception based on Switchable Ionic Polarization

AbstractDiverse touch experiences offer a path toward greater human–machine interaction, which is essential for the development of haptic technology. Recent advances in triboelectricity‐based touch sensors provide great advantages in terms of cost, simplicity of design, and use of a broader range of materials. Since performance solely relies on the level of contact electrification between materials, triboelectricity‐based touch sensors cannot effectively be used to measure the extent of deformation of materials under a given mechanical force. Here, an ion‐doped gelatin hydrogel (IGH)‐based touch sensor is reported to identify not only contact with an object but also deformation under a certain level of force. Switchable ionic polarization of the gelatin hydrogel is found to be instrumental in allowing for different sensing mechanisms when it is contacted and deformed. The results show that ionic polarization relies on conductivity of the hydrogels. Quantitative studies using voltage sweeps demonstrate that higher ion mobility and shorter Debye length serve to improve the performance of the mechanical stimuli‐perceptible sensor. It is successfully demonstrated that this sensor offers dynamic deformation‐responsive signals that can be used to control the motion of a miniature car. This study broadens the potential applications for ionic hydrogel‐based sensors in a human–machine communication system.

Fast CRDNN: Towards on Site Training of Mobile Construction Machines

The CRDNN is a combined neural network that can increase the holistic efficiency of torque based mobile working machines by about 9% by means of accurately detecting the truck loading cycles. On the one hand, it is a robust but offline learning algorithm so that it is more accurate and much quicker than the previous methods. However, on the other hand, its accuracy cannot always be guaranteed because of the diversity of the mobile machines industry and the nature of the offline method. To address the problem, we utilize the transfer learning algorithm and the Internet of Things (IoT) technology. Concretely, the CRDNN is first trained by computer and then saved in the on-board ECU. In case that the pre-trained CRDNN is not suitable for the new machine, the operator can label some new data by our App connected to the on-board ECU of that machine through Bluetooth. With the newly labeled data, we can directly further train the pre-trained CRDNN on the ECU without overloading since transfer learning requires less computation effort than training the networks from scratch. In our paper, we prove this idea and show that CRDNN is always competent, with the help of transfer learning and IoT technology by field experiment, even the new machine may have a different distribution. Also, we compared the performance of other SOTA multivariate time series algorithms on predicting the working state of the mobile machines, which denotes that the CRDNNs are still the most suitable solution. As a by-product, we build up a human-machine communication system to label the dataset, which can be operated by engineers without knowledge about Artificial Intelligence (AI).

Introducing Emotions to the Modelingof Intra- and Inter-Personal Influencesin Parent-Adolescent Conversations

An understanding of the dynamics underlying emotional interactions between speakers is essential to the design of effective conversational strategies for interviews, mental health therapies, teaching and counseling, as well as the design of naturalistic human-machine communication systems. The present study introduces a new approach to the modeling of emotional influences during parent-adolescent conversations. The proposed dynamic influence model (DIM) estimates the joint conditional probabilities of speaker's states as a linear combination of simpler inter- and intra-speaker conditional probabilities. Contrary to the previously existing influence models (IMs), the DIM's coefficients are given not as static, constant values but as dynamically changing functions of the time delay between the current and the previous state. The speaker's states were annotated using four labels (speech with positive emotion, speech with negative emotion, emotionally neutral speech and silence with undefined emotion). Experimental results based on the audio recordings of 63 different naturalistic (not acted) parent-adolescent conversations showed that the proposed method leads to psychologically plausible observations. It was also demonstrated that the proposed DIM can achieve up to 20 percent higher accuracy of discriminating between emotional influence patterns of parents and adolescents when compared to the previously used static IM.

Wavelet network for recognition system of Arabic word

Focusing on the development of new technologies of information, research in the speech communication field is an activity in full expansion. Several disciplines and skills interact in order to improve performance of Human Machine Communication Systems (HMC). In order to increase the performance of these systems, various techniques, including Hidden Markov Models (HMM) and Neural Network (NN), are implemented. In this paper, we advance a new approach for modelling of acoustic units and a new method for speech recognition, especially recognition of Arabic word, adapting to this new type of modelling based on Wavelet Network (WN). The new recognition system is a hybrid classifier. It is based on NN as a general model and the wavelets assume the role of activation function. Our approach of speech recognition is divided into two parts: training, and recognition phases. The training stage is based on audio corpus. After converting all training sig- nals from original format to a specific parameterisation, each acoustic vector will be modelled by WN. These vectors will refine and cover all signal properties in one model. It consists in generating a WN for every training signal. The recognition phase is divided into three steps. The first is to extract features from the input vector to be recognized. The second is to estimate all resulting vectors from training WN.

Audio-visual enhancement of speech in noise.

A key problem for telecommunication or human-machine communication systems concerns speech enhancement in noise. In this domain, a certain number of techniques exist, all of them based on an acoustic-only approach--that is, the processing of the audio corrupted signal using audio information (from the corrupted signal only or additive audio information). In this paper, an audio-visual approach to the problem is considered, since it has been demonstrated in several studies that viewing the speaker's face improves message intelligibility, especially in noisy environments. A speech enhancement prototype system that takes advantage of visual inputs is developed. A filtering process approach is proposed that uses enhancement filters estimated with the help of lip shape information. The estimation process is based on linear regression or simple neural networks using a training corpus. A set of experiments assessed by Gaussian classification and perceptual tests demonstrates that it is indeed possible to enhance simple stimuli (vowel-plosive-vowel sequences) embedded in white Gaussian noise.

Deployment of human-machine dialogue systems.

The deployment of systems for human-to-machine communication by voice requires overcoming a variety of obstacles that affect the speech-processing technologies. Problems encountered in the field might include variation in speaking style, acoustic noise, ambiguity of language, or confusion on the part of the speaker. The diversity of these practical problems encountered in the "real world" leads to the perceived gap between laboratory and "real-world" performance. To answer the question "What applications can speech technology support today?" the concept of the "degree of difficulty" of an application is introduced. The degree of difficulty depends not only on the demands placed on the speech recognition and speech synthesis technologies but also on the expectations of the user of the system. Experience has shown that deployment of effective speech communication systems requires an iterative process. This paper discusses general deployment principles, which are illustrated by several examples of human-machine communication systems.

Direct Finger Pointing as a Man-Machine Interface

Performance in using a finger to point at an item on a screen is evaluated in three ex-periments. Taking into account the performance evaluation, an experiment system using a pointing gesture is designed and demonstrated. In preparing the experiments, pointing gestures are classified into three types by the user’s intensity of gaze. Through analyses of the three experiments, performance models based on the error rate and the pointing time are proposed. The error rate is derived from the normal distribution of the error-angle defined by the observed direction and the correct direction. The pointing time is found to be in accordance with Fitts’s law. By using the performance models, an experiment system, called “Illustrated Map Guidance System”, is designed. In this system, a user can ask the name of an item on a map, such as a building, a station or a park, using an intuitively natural pointing gesture and language. The result of the experiments and the prototyping should contribute to advanced human-machine communication systems such as computer aided design and control.

Constraints on the perception of synthetic speech generated by rule

Within the next few years, there will be an extensive proliferation of various types of voice response devices in human-machine communication systems. Unfortunately, at present, relatively little basic or applied research has been carried out on the intelligibility, comprehension, and perceptual processing of synthetic speech produced by these devices. On the basis of our research, we identify five factors that must be considered in studying the perception of synthetic speech: (1) the specific demands imposed by a particular task, (2) the inherent limitations of the human information processing system, (3) the experience and training of the human listener, (4) the linguistic structure of the message set, and (5) the structure and quality of the speech signal.