Multimodal Feature Representation Research Articles

Analyzing sentiment and its application in deep learning: Consistent behavior across multiple occasions

This article offers a systematic review of the evolution in sentiment analysis techniques, moving from unimodal to multimodal to multi-occasion methodologies, with an emphasis on the integration and application of deep learning in sentiment analysis. Firstly, the paper presents the theoretical foundation of sentiment analysis, including the definition and classification of affect and emotion. It then delves into the pivotal technologies used in unimodal sentiment analysis, specifically within the domains of text, speech, and image analysis, examining feature extraction, representation, and classification models. Subsequently, the focus shifts to multimodal sentiment analysis. The paper offers a survey of widely utilized multimodal sentiment datasets, feature representation and fusion techniques, as well as deep learning-based multimodal sentiment analysis models such as attention networks and graph neural networks. It further addresses the application of these multimodal sentiment analysis techniques in social media, product reviews, and public opinion monitoring. Lastly, the paper underscores that challenges persist in the area of multimodal sentiment fusion, including data imbalance and disparities in feature expression. It calls for further research into cross-modal feature expression, dataset augmentation, and explainable modeling to enhance the performance of complex sentiment analysis across multiple occasions.

Research on an Enhanced Multimodal Network for Specific Emitter Identification

Specific emitter identification (SEI) refers to the task of distinguishing similar emitters, especially those of the same type and transmission parameters, which is one of the most critical tasks of electronic warfare. However, SEI is still a challenging task when a feature has low physical representation. Feature representation largely determines the recognition results. Therefore, this article expects to move toward robust feature representation for SEI. Efficient multimodal strategies have great potential for applications using multimodal data and can further improve the performance of SEI. In this research, we introduce a multimodal emitter identification method that explores the application of multimodal data, time-series radar signals, and feature vector data to an enhanced transformer, which employs a conformer block to embed the raw data and integrates an efficient multimodal feature representation module. Moreover, we employ self-knowledge distillation to mitigate overconfident predictions and reduce intra-class variations. Our study reveals that multimodal data provide sufficient information for specific emitter identification. Simultaneously, we propose the CV-CutMixOut method to augment the time-domain signal. Extensive experiments on real radar datasets indicate that the proposed method achieves more accurate identification results and higher feature discriminability.

MS3Net: a deep ensemble learning approach for ship classification in heterogeneous remote sensing data

ABSTRACT Maritime ship classification is essential for effectively monitoring oceanic activities but faces challenges when using heterogeneous remote sensing data. This research presents a novel dataset called Heterogeneous Ship Data, created by combining synthetic aperture radar (SAR) and optical satellite images of ships. The integration of multimodal data aims to harness the complementary advantages of SAR’s all-weather imaging capabilities and optical data’s higher interretability and shape discrimination. To leverage the heterogeneous dataset, this paper proposes an ensemble deep learning model called MS3Net (Multi-convolutional network using Spatial attention and Second-order pooling for Ship image classification). MS3Net integrates spatial attention mechanisms and second-order pooling. MS3Net has multiple convolutional branches, each tailored to extract distinct representations from the SAR and optical data through varied filter sizes and kernels. The spatial attention units focus the learning on the most informative parts of the image while suppressing irrelevant features. The second-order pooling captures higher-order interactions between feature elements to model complex relationships. A concatenation fusion strategy aggregates the diverse branch outputs into a joint multimodal feature representation for enhanced discrimination. Extensive experiments demonstrate MS3Net’s superior classification accuracy over prior state-of-the-art approaches. The proposed heterogeneous data representation coupled with the attention-based deep ensemble learning model contributes to maritime surveillance by enabling robust ship recognition across diverse sensing modalities.

Multi-modal brain tumor segmentation via disentangled representation learning and region-aware contrastive learning

Brain tumors are threatening the life and health of people in the world. Automatic brain tumor segmentation using multiple MR images is challenging in medical image analysis. It is known that accurate segmentation relies on effective feature learning. Existing methods address the multi-modal MR brain tumor segmentation by explicitly learning a shared feature representation. However, these methods fail to capture the relationship between MR modalities and the feature correlation between different target tumor regions. In this paper, I propose a multi-modal brain tumor segmentation network via disentangled representation learning and region-aware contrastive learning. Specifically, a feature fusion module is first designed to learn the valuable multi-modal feature representation. Subsequently, a novel disentangled representation learning is proposed to decouple the fused feature representation into multiple factors corresponding to the target tumor regions. Furthermore, contrastive learning is presented to help the network extract tumor region-related feature representations. Finally, the segmentation results are obtained using the segmentation decoders. Quantitative and qualitative experiments conducted on the public datasets, BraTS 2018 and BraTS 2019, justify the importance of the proposed strategies, and the proposed approach can achieve better performance than other state-of-the-art approaches. In addition, the proposed strategies can be extended to other deep neural networks.

Contrastive Learning of Multimodal Consistency Feature Representation for Remote Sensing Image Registration

Contrastive Learning of Multimodal Consistency Feature Representation for Remote Sensing Image Registration

SGFNet: Semantic-Guided Fusion Network for RGB-Thermal Semantic Segmentation

Recently, semantic segmentation based on RGB and thermal infrared (TIR) images has become a research hotspot because of its stability in the weak light environment. However, most of the current methods ignore the differences between the two modalities of data and do not use semantic information in multi-modal fusion. In this paper, we propose a novel Semantic-Guided Fusion Network (SGFNet) for RGB-Thermal semantic segmentation, which makes full use of semantic information in the multi-modal fusion. Our SGFNet consists of an asymmetric encoder with TIR branch and RGB branch and a decoder. We concentrate on enhancing the multi-modal feature representation in the encoder with a pattern of fusion and enhancement. Specifically, considering that TIR images are stable under weak light conditions, we first propose a Semantic Guidance Head to extract semantic information in the TIR branch. In the RGB branch, we propose a Multi-modal Coordination and Distillation Unit to fuse multi-modal features first. Then, we propose a Cross-level and Semantic-guided Enhancement Unit to enhance the fused features with cross-level information and semantic information. We arrange these two units at all stages of the RGB branch to generate features with strong representation abilities at different levels. For the decoder, to obtain large receptive fields and fine edges, we improve the Lawin ASPP decoder by introducing edge information extracted from the low-level features, proposing the edge-aware Lawin ASPP decoder. With our encoder and decoder working together, our SGFNet can identify objects accurately and segment objects finely. Extensive experiments on the MFNet dataset demonstrate the superior performance of the proposed SGFNet compared with state-of-the-art methods. The code and results of our method are available at https://github.com/kw717/SGFNet.

GMRLNet: A graph-based manifold regularization learning framework for placental insufficiency diagnosis on incomplete multimodal ultrasound data.

Multimodal analysis of placental ultrasound (US) and microflow imaging (MFI) could greatly aid in the early diagnosis and interventional treatment of placental insufficiency (PI), ensuring a normal pregnancy. Existing multimodal analysis methods have weaknesses in multimodal feature representation and modal knowledge definitions and fail on incomplete datasets with unpaired multimodal samples. To address these challenges and efficiently leverage the incomplete multimodal dataset for accurate PI diagnosis, we propose a novel graph-based manifold regularization learning (MRL) framework named GMRLNet. It takes US and MFI images as input and exploits their modality-shared and modality-specific information for optimal multimodal feature representation. Specifically, a graph convolutional-based shared and specific transfer network (GSSTN) is designed to explore intra-modal feature associations, thus decoupling each modal input into interpretable shared and specific spaces. For unimodal knowledge definitions, graph-based manifold knowledge is introduced to describe the sample-level feature representation, local inter-sample relations, and global data distribution of each modality. Then, an MRL paradigm is designed for inter-modal manifold knowledge transfer to obtain effective cross-modal feature representations. Furthermore, MRL transfers the knowledge between both paired and unpaired data for robust learning on incomplete datasets. Experiments were conducted on two clinical datasets to validate the PI classification performance and generalization of GMRLNet. State-of-the-art comparisons show the higher accuracy of GMRLNet on incomplete datasets. Our method achieves 0.913 AUC and 0.904 balanced accuracy (bACC) for paired US and MFI images, as well as 0.906 AUC and 0.888 bACC for unimodal US images, illustrating its application potential in PI CAD systems.

MBFRDH: Design of a Multimodal Bioinspired Feature Representation Deep Learning Model for Identification of Heart-Diseases

Electrocardiograms (ECGs) are generated by checking different beating patterns of heart, and are widely used for identification of multiple heart-related issues. Existing deep learning models that are proposed for ECG analysis are either highly complex, or showcase lower scalability when applied to clinical scans. To overcome these issues, this text proposes design of a novel multimodal bioinspired feature representation deep learning model for identification of heart-diseases. The proposed model initially collects large-scale ECG datasets, and extracts Fourier, Cosine, iVector, Gabor, and Wavelet components. These components are given to a Grey Wolf Optimization (GWO) based feature selection model, which assists in identification of high-inter-class variance feature sets. This is done via modelling a variance-based fitness function and fusing it with an Iterative Learning Model (ILM) that use feedback-accuracy levels for optimization of selected feature sets. The extracted features are used to incrementally train a custom 1D Binary-Augmented Convolutional Neural Network (1D BACNN) that can be trained for multiclass scenarios. The BACNN Model is trained individually for each of the heart diseases. Each BACNN categorizes input ECG samples between ‘Normal’, and ‘Heart-Disease’ categories. Due to use of this binary-type classification, the proposed model is able to achieve a consistent 99.9% accuracy for multiple heart disease sets, which is found to be higher than most of the existing multiclass techniques. The model was tested for Angina, Arrhythmia, Valve disease, and Congenital heart conditions, and was observed to achieve 3.5% higher precision, 4.9% higher accuracy, with 1.2% increase is computational delay, which makes it highly suitable for real-time clinical use cases.

Contrastive self-supervised representation learning without negative samples for multimodal human action recognition.

Action recognition is an important component of human-computer interaction, and multimodal feature representation and learning methods can be used to improve recognition performance due to the interrelation and complementarity between different modalities. However, due to the lack of large-scale labeled samples, the performance of existing ConvNets-based methods are severely constrained. In this paper, a novel and effective multi-modal feature representation and contrastive self-supervised learning framework is proposed to improve the action recognition performance of models and the generalization ability of application scenarios. The proposed recognition framework employs weight sharing between two branches and does not require negative samples, which could effectively learn useful feature representations by using multimodal unlabeled data, e.g., skeleton sequence and inertial measurement unit signal (IMU). The extensive experiments are conducted on two benchmarks: UTD-MHAD and MMAct, and the results show that our proposed recognition framework outperforms both unimodal and multimodal baselines in action retrieval, semi-supervised learning, and zero-shot learning scenarios.

A Discriminant Information Theoretic Learning Framework for Multi-modal Feature Representation

As sensory and computing technology advances, multi-modal features have been playing a central role in ubiquitously representing patterns and phenomena for effective information analysis and recognition. As a result, multi-modal feature representation is becoming a progressively significant direction of academic research and real applications. Nevertheless, numerous challenges remain ahead, especially in the joint utilization of discriminatory representations and complementary representations from multi-modal features. In this article, a discriminant information theoretic learning (DITL) framework is proposed to address these challenges. By employing this proposed framework, the discrimination and complementation within the given multi-modal features are exploited jointly, resulting in a high-quality feature representation. According to characteristics of the DITL framework, the newly generated feature representation is further optimized, leading to lower computational complexity and improved system performance. To demonstrate the effectiveness and generality of DITL, we conducted experiments on several recognition examples, including both static cases, such as handwritten digit recognition, face recognition, and object recognition, and dynamic cases, such as video-based human emotion recognition and action recognition. The results show that the proposed framework outperforms state-of-the-art algorithms.

Multimodal negative sentiment recognition of online public opinion on public health emergencies based on graph convolutional networks and ensemble learning

To improve the effect of multimodal negative sentiment recognition of online public opinion on public health emergencies, we constructed a novel multimodal fine-grained negative sentiment recognition model based on graph convolutional networks (GCN) and ensemble learning. This model comprises BERT and ViT-based multimodal feature representation, GCN-based feature fusion, multiple classifiers, and ensemble learning-based decision fusion. Firstly, the image-text data about COVID-19 is collected from Sina Weibo, and the text and image features are extracted through BERT and ViT, respectively. Secondly, the image-text fused features are generated through GCN in the constructed microblog graph. Finally, AdaBoost is trained to decide the final sentiments recognized by the best classifiers in image, text, and image-text fused features. The results show that the F1-score of this model is 84.13% in sentiment polarity recognition and 82.06% in fine-grained negative sentiment recognition, improved by 4.13% and 7.55% compared to the optimal recognition effect of image-text feature fusion, respectively.

A novel extended multimodal AI framework towards vulnerability detection in smart contracts

Current automatic data-driven vulnerability detection in smart contracts selects and processes features of interest under black box settings without empirical justification. In this paper, we propose a smart contract testing methodology that bestows developers with flexible, practical and customizable strategies to detect vulnerabilities. Our work enforces strong whitebox knowledge to a series of supervised multimodal tasks under static analysis. Each task encapsulates a vulnerability detection branch test and pipelines feature selection, dimension unification, feature fusion, model training and decision-making. We exploit multiple features made up of code and graph embeddings at the single modality level (intramodal settings) and across individual modalities (intermodal settings). We assign each task to either intramodal or intermodal settings, and show how to train state-of-the-art self-attentive bi-LSTM, textCNN, and random forest (RF) models to extract a joint multimodal feature representation per task. We evaluate our framework over 101,082 functions extracted from the SmartEmbed dataset, and rank each multimodal vulnerability mining strategy in terms of detection performance. Extensive experiments show that our work outperforms existing schemes, and the highest performance reaches 99.71%.

Multimodal spatial-temporal feature representation and its application in action recognition

目的 在人体行为识别研究中，利用多模态方法将深度数据与骨骼数据相融合，可有效提高动作的识别率。针对深度图像信息数据量大、冗余度高等问题，提出一种通过获取关键时程信息动作帧序列降低冗余的算法，即质心运动路径松弛算法，并根据不同模态数据的特点，提出一种新的时空特征表示方法。方法 质心运动路径松弛算法根据质心在相邻帧之间的运动距离，计算图像差分后获得的活跃部分的相似系数，然后剔除掉相似度高的帧，获得足以表达行为的关键时程信息。根据图像动态部分的变化特性、人体各部分在运动中的协同性和局部显著性特征构建一种新的时空特征表示方法。结果 在MSR-Action3D数据集上对本文方法的效果进行验证。在3个子集中进行交叉验证的平均分类识别率为95.743 2%，分别比Multi-fused，CovP3DJ，D3D-LSTM（densely connected 3DCNN and long short-term memory），Joint Subset Selection方法高2.443 2%，4.763 2%，0.343 2%，0.213 2%。本文方法在使用完整数据集的扩展实验中进行交叉验证的分类识别率为93.040 3%，具有很好的鲁棒性。结论 实验结果表明，本文提出的去冗余算法在降低冗余后提升了识别效果，提取的特征之间具有相关性低的特点，在组合识别中具有良好的互补性，有效提高了分类识别的精确度。;Objective Human body motion-related recognition has been developing in the context of computer vision and pattern recognition like auxiliary human-computer interaction，motion analysis，intelligent monitoring，and virtual reality. To obtain two-dimensional information for its behavioral recognition，conventional motion behavior recognition is mainly used the RGB image sequence captured by RGB camera. To improve the ability to detect short-duration fragments，current feature descriptors for RGB image sequences are employed to characterize human behavior，such as histogram of oriented gradient（HOG），histogram of optical flow （HOF），and a three-dimensional feature pyramid. Some researchers are focused on the feature that image depth is insensitive to ambient light since RGB images are oriented to behavior image sequences of objects in terms of two-dimensional information. The depth information of the image is coordinated with the features of RGB image to describe the related behavior. Human behavior recognition-relevant multi-modal method can be used to fuse depth data and skeleton data，which can improve the recognition rate of action effectively. Recent depth map is widely used in relevant to human behavior recognition. But，the collection of depth information data is required to be optimized because of time complexity of feature extraction and space complexity of feature storage. To resolve the problems，we develop an algorithm to optimize frames of the depth map and resource consumption. At the same time，a new representation of motion features is facilitated as well according to the motion information of the centroid. Method First，the temporal feature vector is used in terms of depth map sequence-extracted time sequence information. The centroid motion path relaxation algorithm is used to realize depth image de-duplication and de redundancy，and the skeleton map-extracted spatial structure feature vector from are spliced to form the spatio-temporal feature input. Next，spatial features are extracted in terms of the original skeleton points coordinates-spliced three-channel spatial feature map. Finally，the fusion probability of spatio-temporal features and spatial features is used for classification and recognition. Our centroid motion path relaxation algorithm is focused on the optimization of redundant information，the time complexity of feature extraction，and the space complexity of feature storage. For the skeleton data，the global feature of motion direction is proposed to fully reflect the integrity and coordination of limb movements. The extracted features are concatenated to obtain the spatio-temporal feature vector，and they can be fused and enhanced through the original coordinates of skeleton points-built three-channel spatial feature map. Its effectiveness is verified on the MSR-Action3D dataset. Result The experimental setting 1 demonstrate that it is 0. 826 0% higher than the depth motion map（DMM）-local binary pattern（LBP）algorithm，1. 015 2% higher than DMM-CRC（collaborative representation classifier），3. 450 1% higher than gradient local auto correlation（DMM-GLAC） algorithm，0. 605 8% higher than EigenJoint algorithm，and 0. 605 8% higher than space-time auto correlation of gradient （STACOG）algorithm is 10. 624 5% higher. After removing redundancy，the result of experimental setting 1 is 0. 126 1% higher as well. The cross-validation on experimental setting 2 show that the average classification and recognition rate in the three subsets is 95. 743 2%，2. 443 2% higher than multi-fused method，4. 763 2% higher than CovP3DJ method，0. 343 2% higher than D3D-LSTM method，and 0. 213 2% higher than joint subset selection method. For the overall data set，it is 2. 030 3% higher than low latency method，0. 240 3% higher than combination of deep models method，and 2. 340 3% higher than complex network coding method. The experimental setting 2 illustrates that the average classification recognition rate of cross-validation in three subsets is 95. 743 2%，and the classification recognition rate of the complete dataset is 93. 040 3%. Conclusion Our algorithm proposed can improve the recognition effect based on redundancy-optimized，and the featuresextracted have lower correlation mutually，which can improve the accuracy of classification recognition effectively.

View-Aware Geometry-Structure Joint Learning for Single-View 3D Shape Reconstruction.

Reconstructing a 3D shape from a single-view image using deep learning has become increasingly popular recently. Most existing methods only focus on reconstructing the 3D shape geometry based on image constraints. The lack of explicit modeling of structure relations among shape parts yields low-quality reconstruction results for structure-rich man-made shapes. In addition, conventional 2D-3D joint embedding architecture for image-based 3D shape reconstruction often omits the specific view information from the given image, which may lead to degraded geometry and structure reconstruction. We address these problems by introducing VGSNet, an encoder-decoder architecture for view-aware joint geometry and structure learning. The key idea is to jointly learn a multimodal feature representation of 2D image, 3D shape geometry and structure so that both geometry and structure details can be reconstructed from a single-view image. To this end, we explicitly represent 3D shape structures as part relations and employ image supervision to guide the geometry and structure reconstruction. Trained with pairs of view-aligned images and 3D shapes, the VGSNet implicitly encodes the view-aware shape information in the latent feature space. Qualitative and quantitative comparisons with the state-of-the-art baseline methods as well as ablation studies demonstrate the effectiveness of the VGSNet for structure-aware single-view 3D shape reconstruction.

Multi-modal knowledge graphs representation learning via multi-headed self-attention

Traditional knowledge graphs (KG) representation learning focuses on the link information between entities, and the effectiveness of learning is influenced by the complexity of KGs. Considering a multi-modal knowledge graph (MKG), due to the introduction of considerable other modal information(such as images and texts), the complexity of KGs further increases, which degrades the effectiveness of representation learning. To resolve this solve the problem, this study proposed the multi-modal knowledge graphs representation learning via multi-head self-attention (MKGRL-MS) model, which improved the effectiveness of link prediction by adding rich multi-modal information to the entity. We first generated a single-modal feature vector corresponding to each entity. Then, we used multi-headed self-attention to obtain the attention degree of different modal features of entities in the process of semantic synthesis. In this manner, we learned the multi-modal feature representation of entities. New knowledge representation is the sum of traditional knowledge representation and an entity’s multi-modal feature representation. Simultaneously, we successfully train our model on two existing models and two different datasets and verified its versatility and effectiveness on the link prediction task.

Robust Transcoding Sensory Information With Neural Spikes.

Neural coding, including encoding and decoding, is one of the key problems in neuroscience for understanding how the brain uses neural signals to relate sensory perception and motor behaviors with neural systems. However, most of the existed studies only aim at dealing with the continuous signal of neural systems, while lacking a unique feature of biological neurons, termed spike, which is the fundamental information unit for neural computation as well as a building block for brain-machine interface. Aiming at these limitations, we propose a transcoding framework to encode multi-modal sensory information into neural spikes and then reconstruct stimuli from spikes. Sensory information can be compressed into 10% in terms of neural spikes, yet re-extract 100% of information by reconstruction. Our framework can not only feasibly and accurately reconstruct dynamical visual and auditory scenes, but also rebuild the stimulus patterns from functional magnetic resonance imaging (fMRI) brain activities. More importantly, it has a superb ability of noise immunity for various types of artificial noises and background signals. The proposed framework provides efficient ways to perform multimodal feature representation and reconstruction in a high-throughput fashion, with potential usage for efficient neuromorphic computing in a noisy environment.

A Discriminative Vectorial Framework for Multi-Modal Feature Representation

Due to the rapid advancements of sensory and computing technology, multi-modal data sources that represent the same pattern or phenomenon have attracted growing attention. As a result, finding means to explore useful information from these multi-modal data sources has quickly become a necessity. In this paper, a discriminative vectorial framework is proposed for multi-modal feature representation in knowledge discovery by employing multi-modal hashing (MH) and discriminative correlation maximization (DCM) analysis. Specifically, the proposed framework is capable of minimizing the semantic similarity among different modalities by MH and exacting intrinsic discriminative representations across multiple data sources by DCM analysis jointly, enabling a novel vectorial framework of multi-modal feature representation. Moreover, the proposed feature representation strategy is analyzed and further optimized based on canonical and non-canonical cases, respectively. Consequently, the generated feature representation leads to effective utilization of the input data sources of high quality, producing improved, sometimes quite impressive, results in various applications. The effectiveness and generality of the proposed framework are demonstrated by utilizing classical features and deep neural network (DNN) based features with applications to image and multimedia analysis and recognition tasks, including data visualization, face recognition, object recognition; cross-modal (text-image) recognition and audio emotion recognition. Experimental results show that the proposed solutions are superior to state-of-the-art statistical machine learning (SML) and DNN algorithms.

Multi-Modal Residual Perceptron Network for Audio-Video Emotion Recognition.

Emotion recognition is an important research field for human–computer interaction. Audio–video emotion recognition is now attacked with deep neural network modeling tools. In published papers, as a rule, the authors show only cases of the superiority in multi-modality over audio-only or video-only modality. However, there are cases of superiority in uni-modality that can be found. In our research, we hypothesize that for fuzzy categories of emotional events, the within-modal and inter-modal noisy information represented indirectly in the parameters of the modeling neural network impedes better performance in the existing late fusion and end-to-end multi-modal network training strategies. To take advantage of and overcome the deficiencies in both solutions, we define a multi-modal residual perceptron network which performs end-to-end learning from multi-modal network branches, generalizing better multi-modal feature representation. For the proposed multi-modal residual perceptron network and the novel time augmentation for streaming digital movies, the state-of-the-art average recognition rate was improved to 91.4% for the Ryerson Audio–Visual Database of Emotional Speech and Song dataset and to 83.15% for the Crowd-Sourced Emotional Multi Modal Actors dataset. Moreover, the multi-modal residual perceptron network concept shows its potential for multi-modal applications dealing with signal sources not only of optical and acoustical types.

A Non-Blind Deconvolution Semi Pipelined Approach to Understand Text in Blurry Natural Images for Edge Intelligence

Text understanding from natural scene images has progressively gained much interest in computer vision due to the frequent emergence of handheld or wearable devices. Sometimes, these devices capture low-quality images. As a result, the image can be disrupted with unpredictable blur caused by text movement and camera shake. Further, the problem is integrated with edge intelligence, an emerging paradigm that drives the computing services, applications, and tasks from mainstream cloud to network edge. The purpose is to minimize the bandwidth cost and latency and improves privacy. Hence, this research introduces a novel semi pipelined technique to address such challenging issue for edge intelligence. The fundamental contributions of this work are as given: (i) after text image enhancement, the natural text images are blurred to create a synthetic dataset; (ii) we introduced image-based 2D Radix-4 DIT FFT and the inverse to deblur the blurred images; (iii) after that, text understanding process is applied on the recovered images. Firstly, the text localization and segmentation are taken out using a novel open contour and 4-connected edge-based region approach. Secondly, the recovered images are classified into text and non-text classes adopting multimodal feature representation. Thirdly, Character Labeling Convolution Neural Network (CL-CNN) model is introduced for character labeling by extracting deep features to work fine on discriminative and ambiguous text. Finally, the experiments validated that the proposed framework achieved promising results on ICDAR 2003, SVT, and IIIT5K compared with standard techniques substantially and from blur images efficiently and flexibly.

Hierarchical fusion of visual and physiological signals for emotion recognition

Emotion recognition is an attractive and essential topic in image and signal processing. In this paper, we propose a multi-level fusion method to combine visual information and physiological signals for emotion recognition. For visual information, we propose a serial fusion of two-stage features to enhance the representation of facial expression in a video sequence. We propose to integrate the Neural Aggregation Network with Convolutional Neural Network feature map to reinforce the vital emotional frames. For physiological signals, we propose a parallel fusion scheme to widen the band of the annotation of the electroencephalogram signals. We extract the frequency feature with the Linear-Frequency Cepstral Coefficients and enhance it with the signal complexity denoted by Sample Entropy (SampEn). In the classification stage, we realize both feature level and decision level fusion of both visual and physiological information. Experimental results validate the effectiveness of the proposed multi-level multi-modal feature representation method.