Time-synchronized sentiment labeling via autonomous online comments data mining: A multimodal information fusion on large-scale multimedia data

A Brain-Inspired Decision-Making method for upper limb exoskeleton based on Multi-Brain-Region structure and multimodal information fusion

Information fusion and multi-classifier system for miner fatigue recognition in plateau environments based on electrocardiography and electromyography signals

This paper promoted the development of new media art and film and television culture creation through multi-modal information fusion and analysis, and discussed the existing problems of new media art and film and television culture creation at present, including piracy, management problems and lack of innovation ability. The network structure of RNN neural network can cycle information among neurons, retain the memory of previous user information in the progressive learning sequence, analyze user behavior data through previous memory, accurately recommend users, and provide artists with a basis for user preferences. The viewing experience scores for works 1 to 5 created using traditional creative methods were 6.23, 6.02, 6.56, 6.64, and 6.88, respectively. The viewing experience scores for works 1 to 5 created through multi-modal information fusion and analysis were 9.41, 9.08, 9.11, 9.61, and 8.44, respectively. Movies created through multi-modal information fusion and analysis had higher viewing experience ratings. The results of this article emphasize that multi-modal information fusion and analysis can overcome the limitations of traditional single creative methods, provide rich and diverse expressions, and enable creators to more flexibly respond to complex creative needs, thereby achieving better creative effects.

We first introduce the concepts of single-modal information processing and multi-modal information fusion in cognitive science. Some classical multi-modal information fusion models and their computer implementations in history are also explained. Under the conditions that each channels information can be obtained, and their features could be unified representation synchronously, the fusion of multi-modal information can be transformed into classification or regression problems. For practical human-computer interaction systems, the performance of multi-modal information fusion largely relies on the accuracy of the single-modal information identification and the design of the interactive system. We present a practical example of multi-modal information fusion system, and discuss its performances on human computer interaction. Finally, the possible and important development trends for multi-modal human-computer interaction techniques and systems are discussed.

The so-called multimodal information refers to the information from different information sources on different or the same side of the same description target. These pieces of information are different in terms of storage structure, representation, semantic connotation, credibility, and emphasis, but there is a certain inevitable connection between them. This paper aims to study how to analyze and study the low carbon of enterprises with the help of multimodal information fusion based on the background of big data and construct the evolutionary neural network of the improved adaptive genetic algorithm. This paper puts forward the problem of low carbon in enterprises, which is based on multimodal information fusion, and then elaborates on the concept and related algorithms of multimodal information fusion. Information fusion has carried out case design and analysis. Through the research and analysis of enterprise low carbon and self-adaptive algorithm, it can be obtained that the neural network has reached the threshold of 3.95 after iterating for nearly 60 generations, and stopped iterating to obtain the best individual. The evolutionary neural network in this paper reaches stability after a small number of iterations and can basically achieve a certain low carbonization.

Cloud computing techniques take the form of distributed computing by utilizing multiple computers to execute computing simultaneously on the service side. To process the increasing quantity of multimedia data, numerous large-scale multimedia data storage computing techniques in the cloud computing have been developed. Of all the techniques, Hadoop plays a key role in the cloud computing. Hadoop, a computing cluster formed by low-priced hardware, can conduct the parallel computing of petabytes of multimedia data. Hadoop features high-reliability, high-efficiency, and high-scalability. The numerous large-scale multimedia data computing techniques include not only the key core techniques, Hadoop and MapReduce, but also the data collection techniques, such as File Transfer Protocol and Flume. In addition, distributed system configuration allocation, automatic installation, and monitoring platform building and management techniques are all included. As a result, only with the integration of all the techniques, a reliable large-scale multimedia data platform can be offered. In this paper, we introduce how cloud computing can make a breakthrough by proposing a multimedia social network dataset on Hadoop platform and implementing a prototype version. Detailed specifications and design issues are discussed as well. An important finding of this article is that we can save more time if we conduct the multimedia social networking analysis using Cloud Hadoop Platform rather than using a single computer. The advantages of cloud computing over the traditional data processing practices are fully demonstrated in this article. The applicable framework designs and the tools available for the large-scale data processing are also proposed. We show the experimental multimedia data including data sizes and processing time.

The high complexity and uncertainty of financial markets make it difficult for predictive models that rely on a single data source to effectively address these challenges. Multimodal information fusion, by integrating heterogeneous data sources such as text, numerical data, images, and time series, reveals hidden connections and opens new avenues for improving the accuracy and robustness of financial forecasts. This paper systematically analyzes the theoretical foundations, key technical approaches, and typical scenarios for applying multimodal information fusion to financial forecasting. The study first analyzes the characteristics and categories of multimodal financial data, encompassing structured market data, unstructured text, time series data streams, and visual information. Secondly, it focuses on the core technologies of multimodal fusion, covering the data layer, feature layer, decision layer, and hybrid fusion strategies. It also analyzes the advantages of deep learning models in processing and fusing heterogeneous data. Through specific application cases such as stock price trend forecasting, credit risk assessment, and macroeconomic indicator forecasting, this paper demonstrates the significant effectiveness of multimodal fusion in capturing market sentiment, identifying potential risks, and improving forecasting accuracy. The study also identifies key challenges currently faced, including data heterogeneity, model interpretability, computational efficiency, and privacy protection. Finally, we outline future research directions, emphasizing the importance of cross-modal alignment, adaptive fusion mechanisms, interpretability enhancement, and privacy-preserving fusion frameworks. This study provides theoretical support and practical references for deepening the application of multimodal fusion in intelligent financial decision-making.

In allusion to the complex requirement of query, a new video retrieval model based on multimodal information fusion was brought forward in this paper. It included multi-models like text retrieval, image query, semantic features extraction, and used relational algebra expression to fuse these multimodal information. Experimental results demonstrate that our method could fully utilize the advantages of multimodal information fusion based on relational expression in video retrieval, and achieve good performance on complex semantic video retrieval.

A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition

The emotional activities of the human body are mainly regulated by the autonomic nervous system, the central nervous system, and the advanced cognition of the human brain. This paper proposes an emotional state recognition method in pilot training tasks based on multimodal information fusion. A set of emotion perception recognition systems, a two-dimensional valence-arousal emotional model, and a multimodal information intelligent perception model were established based on the human-computer interaction mode during flight training; an intelligent perception system was designed to collect and intelligently perceive four kinds of peripheral physiological signals of pilots in real-time. And based on traditional machine learning models, a binary tree support vector machine was designed to optimize and improve the multimodal information co-integration decision model, which increased the accuracy of emotional state recognition in flight training by 37.58% on average. The experimental result showed that it realizes accurate monitoring and identification of real-time emotional state, helps to improve the effect of flight training and flight safety, maintains the efficiency of operation, and has important research significance and application prospects in the field of pilot training.

Understanding human emotional states is indispensable for our daily interaction, and we can enjoy more natural and friendly human computer interaction (HCI) experience by fully utilizing human’s affective states. In the application of emotion recognition, multimodal information fusion is widely used to discover the relationships of multiple information sources and make joint use of a number of channels, such as speech, facial expression, gesture and physiological processes. This thesis proposes a new framework of emotion recognition using information fusion based on the estimation of information entropy. The novel techniques of information theoretic learning are applied to feature level fusion and score level fusion. The most critical issues for feature level fusion are feature transformation and dimensionality reduction. The existing methods depend on the second order statistics, which is only optimal for Gaussian-like distributions. By incorporating information theoretic tools, a new feature level fusion method based on kernel entropy component analysis is proposed. For score level fusion, most previous methods focus on predefined rule based approaches, which are usually heuristic. In this thesis, a connection between information fusion and maximum correntropy criterion is established for effective score level fusion. Feature level fusion and score level fusion methods are then combined to introduce a two-stage fusion platform. The proposed methods are applied to audiovisual emotion recognition, and their effectiveness is evaluated by experiments on two publicly available audiovisual emotion databases. The experimental results demonstrate that the proposed algorithms achieve improved performance in comparison with the existing methods. The work of this thesis offers a promising direction to design more advanced emotion recognition systems based on multimodal information fusion and has great significance to the development of intelligent human computer interaction systems.

Understanding human emotional states is indispensable for our daily interaction, and we can enjoy more natural and friendly human computer interaction (HCI) experience by fully utilizing human’s affective states. In the application of emotion recognition, multimodal information fusion is widely used to discover the relationships of multiple information sources and make joint use of a number of channels, such as speech, facial expression, gesture and physiological processes. This thesis proposes a new framework of emotion recognition using information fusion based on the estimation of information entropy. The novel techniques of information theoretic learning are applied to feature level fusion and score level fusion. The most critical issues for feature level fusion are feature transformation and dimensionality reduction. The existing methods depend on the second order statistics, which is only optimal for Gaussian-like distributions. By incorporating information theoretic tools, a new feature level fusion method based on kernel entropy component analysis is proposed. For score level fusion, most previous methods focus on predefined rule based approaches, which are usually heuristic. In this thesis, a connection between information fusion and maximum correntropy criterion is established for effective score level fusion. Feature level fusion and score level fusion methods are then combined to introduce a two-stage fusion platform. The proposed methods are applied to audiovisual emotion recognition, and their effectiveness is evaluated by experiments on two publicly available audiovisual emotion databases. The experimental results demonstrate that the proposed algorithms achieve improved performance in comparison with the existing methods. The work of this thesis offers a promising direction to design more advanced emotion recognition systems based on multimodal information fusion and has great significance to the development of intelligent human computer interaction systems.

<p>In order to address the issue of multi-information fusion, this paper proposed a method for bearing fault diagnosis based on multisource and multimodal information fusion. Existing bearing fault diagnosis methods mainly rely on single sensor information. Nevertheless, mechanical faults in bearings are intricate and subject to countless excitation disturbances, which poses a great challenge for accurate identification if only relying on feature extraction from single sensor input. In this paper, a multisource information fusion model based on auto-encoder was first established to achieve the fusion of multi-sensor signals. Based on the fused signals, multimodal feature extraction was realized by integrating image features and time-frequency statistical information. The one-dimensional vibration signals were converted into two-dimensional time-frequency images by continuous wavelet transform (CWT), and then they were fed into the Resnet network for fault diagnosis. At the same time, the time-frequency statistical features of the fused 1D signal were extracted from the integrated perspective of time and frequency domains and inputted into the improved 1D convolutional neural network model based on the residual block and attention mechanism (1DCNN-REA) model to realize fault diagnosis. Finally, the tree-structured parzen estimator (TPE) algorithm was utilized to realize the integration of two models in order to improve the diagnostic effect of a single model and obtain the final bearing fault diagnosis results. The proposed model was validated using real experimental data, and the results of the comparison and ablation experiments showed that compared with other models, the proposed model can precisely diagnosis the fault type with an accuracy rate of 98.93%.</p>

According to the characteristics and division rules of the modules, this paper divides the rotary module, the swing module, and the mobile module. In order to realize the rapid identification of modules, according to the basic principles of module design, the above modules are put into the module library established by Access. According to the modular modeling method, kinematic models are established, respectively. In order to automatically establish the kinematic model of the robot, a unified expression of modules is established. According to the unified expression of the modules, the kinematics of the reconfigurable robot is analyzed. According to the characteristics of the configuration plane, the configuration plane is divided, and the expression form of the position and attitude of the configuration plane is given. Combined with the principle of neural network and multimodal information fusion, a multimodal information fusion model based on long- and short-term memory neural network is established. Aiming at the control problem of mechanically driven reconfigurable robots, a specific long- and short-term memory neural network model is designed, and the long- and short-term memory neural network algorithm is applied to the robot control problem based on multimodal information fusion. The design of the controller and driver of each joint is the basis of the distributed control system. This paper discusses the hardware design of the joint controller and driver and the realization of the position control system and discusses the method of realizing distributed control based on the Modbus protocol of RS485 communication. Through the comprehensive experiment of the configuration, the point control and the continuous path control are carried out to verify the correctness of the theoretical analysis of the system and the reliability of the system hardware and software operation.

Continuous health monitoring of wind turbine blades under all-weather and multi-operating conditions represents a significant challenge in the renewable energy sector. In this article, we present a fault diagnosis approach leveraging multi-modal information fusion and deep learning with continuous state division, thereby overcoming the limitations of traditional methods in complex and noisy environments. The operational conditions of wind turbine blades are categorized into two primary states: the wind operation state and the sudden shutdown state. Additionally, various climate types, including sunny, foggy, windy conditions, differing lighting levels, and others, are considered in the analysis. During wind operation, sound and vibration signals exhibit higher efficacy for fault detection; however, high noise levels may introduce interferences. To address the issue of indistinct fault characteristics after deep convolution due to multiple noise factors, which could result in reduced diagnostic accuracy, we propose a robust fast Fourier transform-ResTransNet model. In the shutdown state, vibration and sound data features become less prominent, making image processing techniques advantageous. Nevertheless, diverse climate types can lead to challenges such as low visibility, high noise, and other interferences. Consequently, we design a Swin-Transformer model that integrates infrared thermal imaging and visible light imagery. This model resolves the problem of non-homologous data representation and ensures accurate information interaction under multi-source data fusion. Simulation results confirm that the proposed fault diagnosis method achieves substantial improvements over existing approaches. To validate the practical applicability of our method, we construct a real-world wind turbine operational environment, simulate several common blade fault scenarios, and collect actual vibration, sound, and image data under varying weather conditions. Based on these simulations, we establish a multi-modal information fusion model tailored for different weather types. Furthermore, to facilitate the integration of our research into real-world wind farm operations, we develop a human–computer interface that enables seamless deployment. The corresponding source code is publicly available at https://github.com/midfigher/Humancomputer-interaction .