Motion Capture Dataset Research Articles

ATGT3D: Animatable Texture Generation and Tracking for 3D Avatars

We propose the ATGT3D an Animatable Texture Generation and Tracking for 3D Avatars, featuring the innovative design of the Eye Diffusion Module (EDM) and Pose Tracking Diffusion Module (PTDM), which are dedicated to high-quality eye texture generation and synchronized tracking of dynamic poses and textures, respectively. Compared to traditional GAN and VAE methods, ATGT3D significantly enhances texture consistency and generation quality in animated scenes using the EDM, which produces high-quality full-body textures with detailed eye information using the HUMBI dataset. Additionally, the Pose Tracking and Diffusion Module (PTDM) monitors human motion parameters utilizing the BEAT2 and AMASS mesh-level animatable human model datasets. The EDM, in conjunction with a basic texture seed featuring eyes and the diffusion model, restores high-quality textures, whereas the PTDM, by integrating MoSh++ and SMPL-X body parameters, models hand and body movements from 2D human images, thus providing superior 3D motion capture datasets. This module maintains the synchronization of textures and movements over time to ensure precise animation texture tracking. During training, the ATGT3D model uses the diffusion model as the generative backbone to produce new samples. The EDM improves the texture generation process by enhancing the precision of eye details in texture images. The PTDM involves joint training for pose generation and animation tracking reconstruction. Textures and body movements are generated individually using encoded prompts derived from masked gestures. Furthermore, ATGT3D adaptively integrates texture and animation features using the diffusion model to enhance both fidelity and diversity. Experimental results show that ATGT3D achieves optimal texture generation performance and can flexibly integrate predefined spatiotemporal animation inputs to create comprehensive human animation models. Our experiments yielded unexpectedly positive outcomes.

Exploring Dataset Bias and Scaling Techniques in Multi-Source Gait Biomechanics: An Explainable Machine Learning Approach

Machine learning has become increasingly important in biomechanics. It allows to unveil hidden patterns from large and complex data, which leads to a more comprehensive understanding of biomechanical processes and deeper insights into human movement. However, machine learning models are often trained on a single dataset with a limited number of participants, which negatively affects their robustness and generalizability. Combining data from multiple existing sources provides an opportunity to overcome these limitations without spending more time on recruiting participants and recording new data. It is furthermore an opportunity for researchers who lack the financial requirements or laboratory equipment to conduct expensive motion capture studies themselves. At the same time, subtle interlaboratory differences can be problematic in an analysis, due to the bias that they introduce. In our study, we investigated differences in motion capture datasets in the context of machine learning, for which we combined overground walking trials from four existing studies. Specifically, our goal was to examine whether a machine learning model was able to predict the original data source based on marker and GRF trajectories of single strides, and how different scaling methods and pooling procedures affected the outcome. Layer-wise relevance propagation was applied to understand which factors were influential to distinguish the original data sources. We found that the model could predict the original data source with a very high accuracy (up to >99%), which decreased by about 15 percentage points when we scaled every dataset individually prior to pooling. However, none of the proposed scaling methods could fully remove the dataset bias. Layer-wise relevance propagation revealed that there was not only one single factor that differed between all datasets. Instead, every dataset had its unique characteristics that were picked up by the model. These variables differed between the scaling and pooling approaches but were mostly consistent between trials belonging to the same dataset. Our results show that motion capture data is sensitive even to small deviations in marker placement and experimental setup and that small inter-group differences should not be overinterpreted during data analysis, especially when the data was collected in different labs. Furthermore, we recommend scaling datasets individually prior to pooling them which led to the lowest accuracy. We want to raise awareness that differences in datasets always exist and are recognizable by machine learning models. Researchers should thus think about how these differences might affect their results when combining data from different studies.

Pose‐to‐Motion: Cross‐Domain Motion Retargeting with Pose Prior

AbstractCreating plausible motions for a diverse range of characters is a long‐standing goal in computer graphics. Current learning‐based motion synthesis methods rely on large‐scale motion datasets, which are often difficult if not impossible to acquire. On the other hand, pose data is more accessible, since static posed characters are easier to create and can even be extracted from images using recent advancements in computer vision. In this paper, we tap into this alternative data source and introduce a neural motion synthesis approach through retargeting, which generates plausible motion of various characters that only have pose data by transferring motion from one single existing motion capture dataset of another drastically different characters. Our experiments show that our method effectively combines the motion features of the source character with the pose features of the target character, and performs robustly with small or noisy pose data sets, ranging from a few artist‐created poses to noisy poses estimated directly from images. Additionally, a conducted user study indicated that a majority of participants found our retargeted motion to be more enjoyable to watch, more lifelike in appearance, and exhibiting fewer artifacts. Our code and dataset can be accessed here.

Learning to Play Guitar with Robotic Hands

AbstractPlaying the guitar is a dexterous human skill that poses significant challenges in computer graphics and robotics due to the precision required in finger positioning and coordination between hands. Current methods often rely on motion capture data to replicate specific guitar playing segments, which restricts the range of performances and demands intricate post‐processing. In this paper, we introduce a novel reinforcement learning model that can play the guitar using robotic hands, without the need for motion capture datasets, from input tablatures. To achieve this, we divide the simulation task for playing guitar into three stages. (a): for an input tablature, we first generate corresponding fingerings that align with human habits. (b): based on the generated fingerings as the guidance, we train a neural network for controlling the fingers of the left hand using deep reinforcement learning, and (c): we generate plucking movements for the right hand based on inverse kinematics according to the tablature. We evaluate our method by employing precision, recall, and F1 scores as quantitative metrics to thoroughly assess its performance in playing musical notes. In addition, we conduct qualitative analysis through user studies to evaluate the visual and auditory effects of guitar performance. The results demonstrate that our model excels in playing most moderately difficult and easier musical pieces, accurately playing nearly all notes.

CP decomposition-based algorithms for completion problem of motion capture data

Motion capture (MoCap) technology is an essential tool for recording and analyzing movements of objects or humans. However, MoCap systems frequently encounter the challenge of missing data, stemming from mismatched markers, occlusion, or equipment limitations. Recovery of these missing data is imperative to maintain the reliability and integrity of MoCap recordings. This paper introduces a novel application of the tensor framework for MoCap data completion. We propose three completion algorithms based on the canonical polyadic (CP) decomposition of tensors. The first algorithm utilizes CP decomposition to capture the low-rank structure of the tensor. However, relying only on low-rank assumptions may be insufficient to deal with complex motion data. Thus, we propose two modified CP decompositions that incorporate additional information, SmoothCP and SparseCP decompositions. SmoothCP integrates piecewise smoothness prior, while SparseCP incorporates sparsity prior, each aiming to improve the accuracy and robustness of MoCap data recovery. To compare and evaluate the merit of the proposed algorithms over other tensor completion methods in terms of several evaluation metrics, we conduct numerical experiments with different MoCap sequences from the CMU motion capture dataset.

Generative adversarial networks to create synthetic motion capture datasets including subject and gait characteristics

Resource-intensive motion capture (mocap) systems challenge predictive deep learning applications, requiring large and diverse datasets. We tackled this by modifying generative adversarial networks (GANs) into conditional GANs (cGANs) that can generate diverse mocap data, including 15 marker trajectories, lower limb joint angles, and 3D ground reaction forces (GRFs), based on specified subject and gait characteristics. The cGAN comprised 1) an encoder compressing mocap data to a latent vector, 2) a decoder reconstructing the mocap data from the latent vector with specific conditions and 3) a discriminator distinguishing random vectors with conditions from encoded latent vectors with conditions. Single-conditional models were trained separately for age, sex, leg length, mass, and walking speed, while an additional model (Multi-cGAN) combined all conditions simultaneously to generate synthetic data. All models closely replicated the training dataset (<8.1 % of the gait cycle different between experimental and synthetic kinematics and GRFs), while a subset with narrow condition ranges was best replicated by the Multi-cGAN, producing similar kinematics (<1°) and GRFs (<0.02 body-weight) averaged by walking speeds. Multi-cGAN also generated synthetic datasets and results for three previous studies using reported mean and standard deviation of subject and gait characteristics. Additionally, unseen test data was best predicted by the walking speed-conditional, showcasing synthetic data diversity. The same model also matched the dynamical consistency of the experimental data (32 % average difference throughout the gait cycle), meaning that transforming the gait cycle data to the original time domain yielded accurate derivative calculations. Importantly, synthetic data poses no privacy concerns, potentially facilitating data sharing.

A Motion Capture Dataset on Human Sitting to Walking Transitions

Sit-to-walk (STW) is a crucial daily task that impacts mobility, independence, and thus quality of life. Existing repositories have limited STW data with small sample sizes (n = 10). Hence, this study presents a STW dataset obtained via the time-up-and-go test, for 65 healthy adults across three age groups – young (19–35 years), middle (36–55 years) and older (above 56 years). The dataset contains lower body motion capture, ground reaction force, surface electromyography, inertial measurement unit data, and responses for the knee injury and osteoarthritis outcome score survey. For validation, the within subjects intraclass correlation coefficients for the maximum and minimum lower body joint angles were calculated with values greater than 0.74, indicating good test-retest reliability. The joint angle trajectories and maximum voluntary contractions are comparable with existing literature, matching in overall trends and range. Accordingly, this dataset allows STW biomechanics, executions, and characteristics to be studied across age groups. Biomechanical trajectories of healthy adults serve as a benchmark in assessing neuromusculoskeletal impairments and when designing assistive technology for treatment or rehabilitation.

Predicting Knee Joint Contact Force Peaks During Gait Using a Video Camera or Wearable Sensors

PurposeEstimating loading of the knee joint may be helpful in managing degenerative joint diseases. Contemporary methods to estimate loading involve calculating knee joint contact forces using musculoskeletal modeling and simulation from motion capture (MOCAP) data, which must be collected in a specialized environment and analyzed by a trained expert. To make the estimation of knee joint loading more accessible, simple input predictors should be used for predicting knee joint loading using artificial neural networks.MethodsWe trained feedforward artificial neural networks (ANNs) to predict knee joint loading peaks from the mass, height, age, sex, walking speed, and knee flexion angle (KFA) of subjects using their existing MOCAP data. We also collected an independent MOCAP dataset while recording walking with a video camera (VC) and inertial measurement units (IMUs). We quantified the prediction accuracy of the ANNs using walking speed and KFA estimates from (1) MOCAP data, (2) VC data, and (3) IMU data separately (i.e., we quantified three sets of prediction accuracy metrics).ResultsUsing portable modalities, we achieved prediction accuracies between 0.13 and 0.37 root mean square error normalized to the mean of the musculoskeletal analysis-based reference values. The correlation between the predicted and reference loading peaks varied between 0.65 and 0.91. This was comparable to the prediction accuracies obtained when obtaining predictors from motion capture data.DiscussionThe prediction results show that both VCs and IMUs can be used to estimate predictors that can be used in estimating knee joint loading outside the motion laboratory. Future studies should investigate the usability of the methods in an out-of-laboratory setting.

LiDARCapV2: 3D human pose estimation with human–object interaction from LiDAR point clouds

Human–object interactions in open environments are common in the real world. Estimating 3D human pose from data where objects occlude the human is a challenging task in biometrics. However, existing LiDAR-based human motion capture datasets lack occlusion scenarios between humans and objects. To overcome this limitation, we propose LiDARHuman51M, a new human–object interaction dataset captured by LiDAR in long-range outdoor scene. It includes human motion labels acquired by an IMU system and synchronous RGB images. Additionally, we present an occlusion-aware method, LiDARCapV2, for capturing human motion from LiDAR point clouds under human–object interaction settings. Our key insight is to overcome object interference in human feature extraction by introducing a module called AgNoise-Segment. A noise augmentation strategy introduced in the AgNoise-Segment module alleviates the dependency of the segmentation accuracy on the effectiveness of 3D human pose estimations. Furthermore, we propose a skeleton extraction module that integrates features learned from the AgNoise-Segment module and predicts the skeleton locations. Quantitative and qualitative experiments demonstrate that LiDARCapV2 can capture high-quality 3D human motion under human–object interaction settings. Experiments on the KITTI and Waymo datasets demonstrate that our method can be generalized to real-world open scenarios.

Combining Transformer, Convolutional Neural Network, and Long Short-Term Memory Architectures: A Novel Ensemble Learning Technique That Leverages Multi-Acoustic Features for Speech Emotion Recognition in Distance Education Classrooms

Speech emotion recognition (SER) is a technology that can be applied to distance education to analyze speech patterns and evaluate speakers’ emotional states in real time. It provides valuable insights and can be used to enhance students’ learning experiences by enabling the assessment of their instructors’ emotional stability, a factor that significantly impacts the effectiveness of information delivery. Students demonstrate different engagement levels during learning activities, and assessing this engagement is important for controlling the learning process and improving e-learning systems. An important aspect that may influence student engagement is their instructors’ emotional state. Accordingly, this study used deep learning techniques to create an automated system for recognizing instructors’ emotions in their speech when delivering distance learning. This methodology entailed integrating transformer, convolutional neural network, and long short-term memory architectures into an ensemble to enhance the SER. Feature extraction from audio data used Mel-frequency cepstral coefficients; chroma; a Mel spectrogram; the zero-crossing rate; spectral contrast, centroid, bandwidth, and roll-off; and the root-mean square, with subsequent optimization processes such as adding noise, conducting time stretching, and shifting the audio data. Several transformer blocks were incorporated, and a multi-head self-attention mechanism was employed to identify the relationships between the input sequence segments. The preprocessing and data augmentation methodologies significantly enhanced the precision of the results, with accuracy rates of 96.3%, 99.86%, 96.5%, and 85.3% for the Ryerson Audio–Visual Database of Emotional Speech and Song, Berlin Database of Emotional Speech, Surrey Audio–Visual Expressed Emotion, and Interactive Emotional Dyadic Motion Capture datasets, respectively. Furthermore, it achieved 83% accuracy on another dataset created for this study, the Saudi Higher-Education Instructor Emotions dataset. The results demonstrate the considerable accuracy of this model in detecting emotions in speech data across different languages and datasets.

DTP: learning to estimate full-body pose in real-time from sparse VR sensor measurements

For virtual reality (VR) applications, estimating full-body pose in real-time is becoming increasingly popular. Previous works have reconstructed full-body motion in real time from an HTC VIVE headset and five VIVE Tracker measurements by solving the inverse kinematics (IK) problem. However, an IK solver may yield unnatural poses and shaky motion. This paper introduces Deep Tracker poser (DTP): a method for real-time full-body pose estimation in VR. This task is difficult due to the ambiguous mapping from the sparse measurements to full-body pose. The data obtained from VR sensors is calibrated, normalized and fed into the deep neural networks (DNN). To learn from sufficient data, we propose synthesizing a VR sensor dataset called AMASS-VR from the AMASS, a collection of various motion capture datasets. Furthermore, feet tracking loss is a common problem of VIVE Tracker. To improve the accuracy and robustness of DTP to the occlusion noise, we simulate the occlusion noise by Gaussian random noise. Then we synthesize an occlusion dataset AMASS-OCC and fine-tune DTP on that. We evaluate DTP by comparing with other popular methods in terms of the accuracy and computational cost. The results indicate that DTP outperforms others in terms of the positional error (1.04 cm) and rotational error (4.22 °). The quantitative and qualitative results show that DTP reconstructs accurate and natural full-body pose even under serious feet occlusion, which indicates the superiority of the DTP in modelling the mapping from sparse joint data to the full-body pose.

Validity of muscle activation estimated with predicted ground reaction force in inverse dynamics based musculoskeletal simulation during gait

The inverse dynamics based musculoskeletal simulation needs ground reaction forces (GRF) as an external force input. GRF can be predicted from kinematic data. However, the validity of estimated muscle activation using the predicted GRF has remained unclear. Therefore, the purpose of this study was to determine the validity of estimated muscle activation with predicted GRF in the inverse dynamics based musculoskeletal simulation. To perform musculoskeletal simulations, an open-source motion capture dataset that contains gait data from 50 healthy subjects was used. CusToM was used for the musculoskeletal simulations. Two sets of inverse dynamics and static optimization were performed, one used predicted GRF (PRED) and another used experimentally measured GRF (EXP). Pearson’s correlation was calculated to evaluate the similarity between EMG and estimated muscle activations for both PRED and EXP. To compare PRED and EXP, paired t-tests were used to compare the trial-wise muscle activation similarity and residuals. Relationships between joint moments and residuals were also tested. The overall muscle activation similarity was comparable in PRED (R = 0.477) and EXP (R = 0.475). The residuals were 2–4 times higher in EXP compared to PRED (P < 0.001). The hip flexion–extension moment was correlated to sagittal plane residual moment (R = 0.467). The muscle activations estimated using predicted GRF were comparable to that with measured GRF in the inverse dynamics based musculoskeletal simulation. Prediction of GRF helps to perform musculoskeletal simulations where the force plates are not available.

Lifting by Image – Leveraging Image Cues for Accurate 3D Human Pose Estimation

The "lifting from 2D pose" method has been the dominant approach to 3D Human Pose Estimation (3DHPE) due to the powerful visual analysis ability of 2D pose estimators. Widely known, there exists a depth ambiguity problem when estimating solely from 2D pose, where one 2D pose can be mapped to multiple 3D poses. Intuitively, the rich semantic and texture information in images can contribute to a more accurate "lifting" procedure. Yet, existing research encounters two primary challenges. Firstly, the distribution of image data in 3D motion capture datasets is too narrow because of the laboratorial environment, which leads to poor generalization ability of methods trained with image information. Secondly, effective strategies for leveraging image information are lacking. In this paper, we give new insight into the cause of poor generalization problems and the effectiveness of image features. Based on that, we propose an advanced framework. Specifically, the framework consists of two stages. First, we enable the keypoints to query and select the beneficial features from all image patches. To reduce the keypoints attention to inconsequential background features, we design a novel Pose-guided Transformer Layer, which adaptively limits the updates to unimportant image patches. Then, through a designed Adaptive Feature Selection Module, we prune less significant image patches from the feature map. In the second stage, we allow the keypoints to further emphasize the retained critical image features. This progressive learning approach prevents further training on insignificant image features. Experimental results show that our model achieves state-of-the-art performance on both the Human3.6M dataset and the MPI-INF-3DHP dataset.

Decaf: Monocular Deformation Capture for Face and Hand Interactions

Existing methods for 3D tracking from monocular RGB videos predominantly consider articulated and rigid objects ( e.g. , two hands or humans interacting with rigid environments). Modelling dense non-rigid object deformations in this setting ( e.g. when hands are interacting with a face), remained largely unaddressed so far, although such effects can improve the realism of the downstream applications such as AR/VR, 3D virtual avatar communications, and character animations. This is due to the severe ill-posedness of the monocular view setting and the associated challenges ( e.g. , in acquiring a dataset for training and evaluation or obtaining the reasonable non-uniform stiffness of the deformable object). While it is possible to naïvely track multiple non-rigid objects independently using 3D templates or parametric 3D models, such an approach would suffer from multiple artefacts in the resulting 3D estimates such as depth ambiguity, unnatural intra-object collisions and missing or implausible deformations. Hence, this paper introduces the first method that addresses the fundamental challenges depicted above and that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system. As a pivotal step in its creation, we process the reconstructed raw 3D shapes with position-based dynamics and an approach for non-uniform stiffness estimation of the head tissues, which results in plausible annotations of the surface deformations, hand-face contact regions and head-hand positions. At the core of our neural approach are a variational auto-encoder supplying the hand-face depth prior and modules that guide the 3D tracking by estimating the contacts and the deformations. Our final 3D hand and face reconstructions are realistic and more plausible compared to several baselines applicable in our setting, both quantitatively and qualitatively. https://vcai.mpi-inf.mpg.de/projects/Decaf

Markerless Motion Tracking With Noisy Video and IMU Data.

Marker-based motion capture, considered the gold standard in human motion analysis, is expensive and requires trained personnel. Advances in inertial sensing and computer vision offer new opportunities to obtain research-grade assessments in clinics and natural environments. A challenge that discourages clinical adoption, however, is the need for careful sensor-to-body alignment, which slows the data collection process in clinics and is prone to errors when patients take the sensors home. We propose deep learning models to estimate human movement with noisy data from videos (VideoNet), inertial sensors (IMUNet), and a combination of the two (FusionNet), obviating the need for careful calibration. The video and inertial sensing data used to train the models were generated synthetically from a marker-based motion capture dataset of a broad range of activities and augmented to account for sensor-misplacement and camera-occlusion errors. The models were tested using real data that included walking, jogging, squatting, sit-to-stand, and other activities. On calibrated data, IMUNet was as accurate as state-of-the-art models, while VideoNet and FusionNet reduced mean ± std root-mean-squared errors by 7.6 ± 5.4 ° and 5.9 ± 3.3 °, respectively. Importantly, all the newly proposed models were less sensitive to noise than existing approaches, reducing errors by up to 14.0 ± 5.3 ° for sensor-misplacement errors of up to 30.0 ± 13.7 ° and by up to 7.4 ± 5.5 ° for joint-center-estimation errors of up to 101.1 ± 11.2 mm, across joints. These tools offer clinicians and patients the opportunity to estimate movement with research-grade accuracy, without the need for time-consuming calibration steps or the high costs associated with commercial products such as Theia3D or Xsens, helping democratize the diagnosis, prognosis, and treatment of neuromusculoskeletal conditions.

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions though, it is important to minimize the number of sensors attached to the subject’s body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this article, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet toward the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions.

4D radar simulator for human activity recognition

AbstractMillimetre‐wave radar has been widely used in health monitoring and human activity recognition owing to its improved range resolution and operation in a variety of environmental conditions. With the MIMO antenna array, 4D radar is increasingly employed in autonomous driving, while its application in assisted living is recent and therefore the value added compared to the increase in signal processing and hardware requirements is still an open question. A model for 4D Time‐division multiplexing (TDM) multiple‐input‐multiple‐output (MIMO) frequency‐modulated Continuous wave radar is established using the human activities from the HDM05 motion capture dataset. The simulator produces an end‐to‐end simulation, including four human motions (jumping Jack, kick, punch, and walk), signal time of flight, noise, MIMO signal processing, and classification. Different pre‐processing and point cloud‐based methods are compared to obtain an average classification accuracy of 90% with PointNet. This study simulates a specific 4D TDM MIMO radar configuration to benchmark signal pre‐processing algorithms, which can also assist other researchers to generate range‐Doppler‐time (range‐Doppler time) point cloud data sets for human activities testing different radar configurations, array configurations, and activities saving valuable time in human resources and hardware development before prototyping to assess expected performances.

Speech emotion recognition with light gradient boosting decision trees machine

&lt;p&gt;Speech emotion recognition aims to identify the emotion expressed in the speech by analyzing the audio signals. In this work, data augmentation is first performed on the audio samples to increase the number of samples for better model learning. The audio samples are comprehensively encoded as the frequency and temporal domain features. In the classification, a light gradient boosting machine is leveraged. The hyperparameter tuning of the light gradient boosting machine is performed to determine the optimal hyperparameter settings. As the speech emotion recognition datasets are imbalanced, the class weights are regulated to be inversely proportional to the sample distribution where minority classes are assigned higher class weights. The experimental results demonstrate that the proposed method outshines the state-of-the-art methods with 84.91% accuracy on the emo-DB dataset, 67.72% on the Ryerson audio-visual database of emotional speech and song (RAVDESS) dataset, and 62.94% on the interactive emotional dyadic motion capture (IEMOCAP) dataset.&lt;/p&gt;

Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks

There has been a growing interest in multimodal sentiment analysis and emotion recognition in recent years due to its wide range of practical applications. Multiple modalities allow for the integration of complementary information, improving the accuracy and precision of sentiment and emotion recognition tasks. However, working with multiple modalities presents several challenges, including handling data source heterogeneity, fusing information, aligning and synchronizing modalities, and designing effective feature extraction techniques that capture discriminative information from each modality. This paper introduces a novel framework called “Attention-based Multimodal Sentiment Analysis and Emotion Recognition (AMSAER)” to address these challenges. This framework leverages intra-modality discriminative features and inter-modality correlations in visual, audio, and textual modalities. It incorporates an attention mechanism to facilitate sentiment and emotion classification based on visual, textual, and acoustic inputs by emphasizing relevant aspects of the task. The proposed approach employs separate models for each modality to automatically extract discriminative semantic words, image regions, and audio features. A deep hierarchical model is then developed, incorporating intermediate fusion to learn hierarchical correlations between the modalities at bimodal and trimodal levels. Finally, the framework combines four distinct models through decision-level fusion to enable multimodal sentiment analysis and emotion recognition. The effectiveness of the proposed framework is demonstrated through extensive experiments conducted on the publicly available Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. The results confirm a notable performance improvement compared to state-of-the-art methods, attaining 85% and 93% accuracy for sentiment analysis and emotion classification, respectively. Additionally, when considering class-wise accuracy, the results indicate that the “angry” emotion and “positive” sentiment are classified more effectively than the other emotions and sentiments, achieving 96.80% and 93.14% accuracy, respectively.

Multi-Level Attention-Based Categorical Emotion Recognition Using Modulation-Filtered Cochleagram

Speech emotion recognition is a critical component for achieving natural human–robot interaction. The modulation-filtered cochleagram is a feature based on auditory modulation perception, which contains multi-dimensional spectral–temporal modulation representation. In this study, we propose an emotion recognition framework that utilizes a multi-level attention network to extract high-level emotional feature representations from the modulation-filtered cochleagram. Our approach utilizes channel-level attention and spatial-level attention modules to generate emotional saliency maps of channel and spatial feature representations, capturing significant emotional channel and feature space from the 3D convolution feature maps, respectively. Furthermore, we employ a temporal-level attention module to capture significant emotional regions from the concatenated feature sequence of the emotional saliency maps. Our experiments on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset demonstrate that the modulation-filtered cochleagram significantly improves the prediction performance of categorical emotion compared to other evaluated features. Moreover, our emotion recognition framework achieves comparable unweighted accuracy of 71% in categorical emotion recognition by comparing with several existing approaches. In summary, our study demonstrates the effectiveness of the modulation-filtered cochleagram in speech emotion recognition, and our proposed multi-level attention framework provides a promising direction for future research in this field.