Video Fusion Research Articles

Target Detection Based on Improved Hausdorff Distance Matching Algorithm for Millimeter-Wave Radar and Video Fusion.

The intelligent transportation system (ITS) is inseparable from people’s lives, and the development of artificial intelligence has made intelligent video surveillance systems more widely used. In practical traffic scenarios, the detection and tracking of vehicle targets is an important core aspect of intelligent surveillance systems and has become a hot topic of research today. However, in practical applications, there is a wide variety of targets and often interference factors such as occlusion, while a single sensor is unable to collect a wealth of information. In this paper, we propose an improved data matching method to fuse the video information obtained from the camera with the millimetre-wave radar information for the alignment and correlation of multi-target data in the spatial dimension, in order to address the problem of poor recognition alignment caused by mutual occlusion between vehicles and external environmental disturbances in intelligent transportation systems. The spatio-temporal alignment of the two sensors is first performed to determine the conversion relationship between the radar and pixel coordinate systems, and the calibration on the timeline is performed by Lagrangian interpolation. An improved Hausdorff distance matching algorithm is proposed for the data dimension to calculate the similarity between the data collected by the two sensors, to determine whether they are state descriptions of the same target, and to match the data with high similarity to delineate the region of interest (ROI) for target vehicle detection.

Real-Time Optical-Wireless Video Surveillance System for High Visual-Fidelity Underwater Monitoring

Real-time, dynamic, and visual monitoring of underwater scenes would become an important research topic with the development of underwater wireless optical communication (UWOC), autonomous underwater vehicle, and video fusion technologies. To this end, we developed a UWOC-based 2K real-time digital video surveillance prototype named AquaF-seer. Using a light-emitting diode and an avalanche photodetector based diffused line-of-sight UWOC system, real-time video transmission with a high resolution of 1920 × 1080 pixels is achieved over various channels, i.e., 1.5-m free space channel, pure water channel, pure water channel with 1.53-mL/s bubble-induced turbulence, pure water channel with 42.40-mL/s bubble-induced turbulence, simulated pure sea water channel, and coastal ocean water channel. Moreover, 46-m and 5-m video monitoring are implemented in free space and an outdoor diving pool. It indicates the reliability of the prototype, which is the first step to realize underwater visual monitoring in future human-robot interaction applications.

The Influence of Public Mental Health Based on Artificial Intelligence Technology on the Teaching Effect of Business Administration Major.

As an emerging technology, public mental health based on artificial intelligence engineering has broad application and development prospects in improving teaching effects, realizing high efficiency and intelligence, and improving refined education. The application of public mental health based on artificial intelligence engineering in the practice of the teaching effect of business administration has a very positive significance for accurately discovering the mental health and teaching effect of business administration students. For the sake of improving the teaching effect of business administration major, we come up with a method for the influence of public mental health on the teaching effect of business administration profession based on artificial intelligence engineering. Firstly, we collect the video and audio of the teaching of business administration, conducts data screening and cleaning, and sorts out the video and audio synchronization clips as training data. Secondly, we propose a multimodal feature extraction network and a multimodal fusion network for the extraction and fusion of video and audio clips, respectively. Then, the fully connected network structure is used to evaluate and classify the effect of business administration professional teaching, and use mental health factors for evaluation to avoid human intervention and improve the prediction effect. Finally, through intensive experimental results, we prove that the method raised by us can use artificial intelligence engineering to evaluate the teaching effect of business administration majors from the perspective of public mental health and achieve good experimental results.

Dynamic Fusion of Text, Video and Audio models for Sentiment Analysis

The paper aims to propose a technique for sentiment analysis that combines audio, textual, and visual data collected from customer input on social media sites such as Instagram, Facebook, and Twitter, as well as feedback forms and product evaluations. The feelings of individuals are represented in these online social media reviews. As a result, sentiment research will be essential for helping the growth of enterprises and individuals. The work assigns variable weights using recording technique for different modules and subsequently fusing it. This enables us to determine which model is optimal for sentiment analysis and how each model affects the final results. A system that can extract information about people's feelings regarding a certain product from text, audio, and video. We may utilize model predictions to determine the amount of consumer happiness with the product, enabling the company to take the required steps to enhance the product.

Design and implementation of multimedia video fusion system

With the rapid development of computer network application technology, various video products and video systems are changing with each passing day. The daily internal communication, training and exchange activities mainly use video conference, paperless conference, projection display, large screen splicing display and other video systems for interaction. These systems are developed and run independently and lack system linkage. In addition, due to the disunity of platforms, the system is prone to multiple devices, complex operation and low resource utilization, which increases the operation and maintenance cost. To solve the above problems, a set of multimedia video fusion system is designed to integrate various video systems to achieve unified control of video conference, paperless conference, projection display, large screen display and other systems, effectively support the use of various business scenarios, and achieve the purpose of improving efficiency and reducing costs.

Real-Time Web Map Construction Based on Multiple Cameras and GIS

Previous VideoGIS integration methods mostly used geographic homography mapping. However, the related processing techniques were mainly for independent cameras and the software architecture was C/S, resulting in large deviations in geographic video mapping for small scenes, a lack of multi-camera video fusion, and difficulty in accessing real-time information with WebGIS. Therefore, we propose real-time web map construction based on the object height and camera posture (RTWM-HP for short). We first consider the constraint of having a similar height for each object by constructing an auxiliary plane and establishing a high-precision homography matrix (HP-HM) between the plane and the map; thus, the accuracy of geographic video mapping can be improved. Then, we map the objects in the multi-camera video with overlapping areas to geographic space and perform the object selection with the multi-camera (OS-CDD) algorithm, which includes the confidence of the object, the distance, and the angle between the objects and the center of the cameras. Further, we use the WebSocket technology to design a hybrid C/S and B/S software framework that is suitable for WebGIS integration. Experiments were carried out based on multi-camera videos and high-precision geospatial data in an office and a parking lot. The case study’s results show the following: (1) The HP-HM method can achieve the high-precision geographic mapping of objects (such as human heads and cars) with multiple cameras; (2) the OS-CDD algorithm can optimize and adjust the positions of the objects in the overlapping area and achieve a better map visualization effect; (3) RTWM-HP can publish real-time maps of objects with multiple cameras, which can be browsed in real time through point layers and hot-spot layers through WebGIS. The methods can be applied to some fields, such as person or car supervision and the flow analysis of customers or traffic passengers.

Millimeter-Wave Radar and Video Fusion Vehicle Detection based on Adaptive Kalman Filtering

Road traffic safety has always been the focus of social concern, and the actual road traffic scenes, where only a single sensor is applied cannot cope with the interference brought by complex external factors, which makes vehicle detection extremely challenging. This paper focuses on a vehicle detection algorithm for the fusion of millimeter-wave radar sensor and monocular camera sensor, including the calibration of millimeter-wave radar and camera, the establishment of a temporal fusion model of the two sensors. Finally, the target information obtained from the two sensors is fused with the data using the adaptive Kalman filter fusion algorithm, which can reduce data ambiguity and increase the reliability and validity of the data. Experiments show that the method can overcome the shortcomings of single sensor in target detection, and the obtained target information is more comprehensive compared with the monocular camera detection results.

Real-time visible and near-infrared video fusion: architecture and implementation

Real-time visible and near-infrared video fusion: architecture and implementation

Attribute filter based infrared and visible image fusion

Infrared and visible image fusion is an effective image processing technique to obtain more comprehensive information, which can help people better understand various scenarios. In this paper, a novel infrared and visible image fusion method is proposed which fully considers the attributes of objects in source images. Benefiting from the attribute and the edge-preserving filters, the prominent objects in the infrared source image are effectively extracted. Then, the weight-based Laplacian pyramid fusion strategy is adopted to get more natural fusion results. The experimental results on the public image fusion datasets and a new infrared–visible video fusion dataset show that the proposed method achieves state-of-the-art fusion performance in terms of both visual and objective evaluations. The proposed algorithm is also implemented in an infrared–visible dual sensor system, which demonstrated the practicability of our fusion method.

Flexible architecture for real-time synchronized processing of multimedia signals

Simultaneous processing of multiple video sources requires each pixel in a frame from a video source to be processed synchronously with the pixels at the same spatial positions in corresponding frames from the other video sources. However, simultaneous processing is challenging as corresponding frames from different video signals provided by multiple sources have time-varying delay because of the electrical and mechanical restrictions inside the video sources hardware that cause deviation in the corresponding frame rates. Researchers overcome the aforementioned challenges either by utilizing ready-made video processing systems or designing and implementing a custom system tailored to their specific application. These video processing systems lack flexibility in handling different applications requirements such as the required number of video sources and outputs, video standards, or frame rates of the input/output videos. In this paper, we present a design for a flexible simultaneous video processing architecture that is suitable for various applications. The proposed architecture is upgradeable to deal with multiple video standards, scalable to process/produce a variable number of input/output videos, and compatible with most video processors. Moreover, we present in details the analog/digital mixed-signals and power distribution considerations used in designing the proposed architecture. As a case study application of the proposed flexible architecture, we utilized the architecture for a realization of a simultaneous video processing system that performs video fusion from visible and near-infrared video sources in real time. We make available the source files of the hardware design along with the bill of material (BOM) of the case study to be a reference for researchers who intend to design and implement simultaneous multi-video processing systems.

A Novel Multi-Sensor Fusion Based Object Detection and Recognition Algorithm for Intelligent Assisted Driving

The object detection and recognition algorithm based on the fusion of millimeter-wave radar and high-definition video data can improve the safety of intelligent-driving vehicles effectively. However, due to the different data modalities of millimeter-wave radar and video, how to fuse the two effectively is the key point. The difficulty lies in the data fusion methods such as insufficient adaptability of image distortion in data alignment and coordinate transformation and also the mismatching of information levels of the data to be fused. To solve the problem of data fusion of millimeter wave radar and video, this paper proposes a decision-level fusion method of millimeter-wave radar and high-definition video data based on angular alignment. Specifically, through the joint calibration and approximate interpolation, projected to polar coordinate system, the radar and the camera are angularly aligned in the horizontal direction. Then objects are detected by a deep neural network model from video data, and combined with those detected by radar to make the joint decision. Finally, object detection and recognition task based on the fusion of the two kinds of data is completed. Theoretical analysis and experimental results indicate that the accuracy of the algorithm based on the two data fusion is superior to that of the single detection and recognition algorithm on the basis of millimeter-wave radar or video data.

Extended LDA Based Topic Analysis for Big Sentiment Data

Owing to the massive emergence of multi-modal data related to big sentiment events, the topic analysis of the text cannot accurately reflect the topic distribution of the whole data set. There are some problems such as semantic deficiency and difficulty in integration. Therefore, the extended LDA(eLDA) based topic analysis for big sentiment data is posed. Firstly, the semantic analysis of text and image data is carried out by probability generation method respectively. The semantic correlation between multi-modal data is used for visual topic learning, a visual topic model is established, and the mapping between visual data and text topics is realized. Furthermore, the fusion of text, audio, video and image data is realized, and multimodal theme analysis is carried out accordingly. The experimental results show that the multimodal topic analysis method can effectively obtain the subject words related to the semantics of big sentiment data. The effect of the subject words tracked by the multi-modal data analysis method is better than that of the single text subject words extraction, which can provide basis for emotional analysis of big sentiment events.

Research on Digital Twin Construction and Safety Management Application of Inland Waterway Based on 3D Video Fusion

The digital twin can reconstruct the scenes of real traffic operation of inland waterways in virtual space, giving birth to new ideas for the digital transformation and upgrading of water safety management. Aiming at the problems of video fragmentation, sub-system data separation and untimely emergency response in the current inland waterway safety monitoring, this paper proposes a new method for reconstructing and applying a digital twin of inland waterways based on 3D video fusion which consists of four parts: (1) Combining drone tilt photography and BIM modeling technology to construct a three-dimensional scene model of inland waterways; (2) Proposing a multi-channel video three-dimensional registration and rendering method to integrate in real time discrete surveillance videos of different angles of view with three-dimensional scene model to establish the spatial correlation between different video images in the scene; (3) Integrating multi-source IoT sensing data to form a scene of digital twin of inland waterways; (4) Exploring innovative application methods for water safety management based on digital twins to meet the water safety management requirements and business logic of inland waterways. A certain segment of the Changhu Shen Line was selected for demonstration and application. The application results show that the construction and application of digital twins for inland waterways combined with 3D video reduces the pressure on managers to understand real-time video footage and improves the efficiency of daily monitoring, evidence collection, and emergency response.

Multimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification

Human communication includes rich emotional content, thus the development of multimodal emotion recognition plays an important role in communication between humans and computers. Because of the complex emotional characteristics of a speaker, emotional recognition remains a challenge, particularly in capturing emotional cues across a variety of modalities, such as speech, facial expressions, and language. Audio and visual cues are particularly vital for a human observer in understanding emotions. However, most previous work on emotion recognition has been based solely on linguistic information, which can overlook various forms of nonverbal information. In this paper, we present a new multimodal emotion recognition approach that improves the BERT model for emotion recognition by combining it with heterogeneous features based on language, audio, and visual modalities. Specifically, we improve the BERT model due to the heterogeneous features of the audio and visual modalities. We introduce the Self-Multi-Attention Fusion module, Multi-Attention fusion module, and Video Fusion module, which are attention based multimodal fusion mechanisms using the recently proposed transformer architecture. We explore the optimal ways to combine fine-grained representations of audio and visual features into a common embedding while combining a pre-trained BERT model with modalities for fine-tuning. In our experiment, we evaluate the commonly used CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets for multimodal sentiment analysis. Ablation analysis indicates that the audio and visual components make a significant contribution to the recognition results, suggesting that these modalities contain highly complementary information for sentiment analysis based on video input. Our method shows that we achieve state-of-the-art performance on the CMU-MOSI, CMU-MOSEI, and IEMOCAP dataset.

A fast fusion method for multi-videos with three-dimensional GIS scenes

Techniques for the fusion of real-world videos with virtual scenes are key to the augmentation of three-dimensional (3-D) virtual geographic scenes, which greatly enhances the immersive visual experience. When a 3-D scene is updated dynamically, the existing video projection-based method for real-virtual fusion is generally slow and inefficient, as all rendered objects must be traversed in the new scene to identify the objects to be fused in the user’s new field of view (FOV). To address this issue, a fast, topology-accounting method for multi-video fusion with 3-D geographic information system (GIS) scenes is proposed. First, the topological models for video object and rendered object are constructed, respectively. Second, by using the topological models, a method that considering topological relationships is proposed to realize rapid identification of rendered objects during the dynamic update of 3-D scenes. Finally, real video and 3-D scene data in Tengzhou City were used to validate the method proposed in this paper. The experiments demonstrated that the method is fast and efficient in the fusion of videos with 3-D GIS scenes, and the computational cost of the proposed method is significantly lower than that of the current method. The proposed method is highly viable and robust, facilitating the fusion of videos with virtual environments.

Multispectral Video Fusion for Non-Contact Monitoring of Respiratory Rate and Apnea.

Continuous monitoring of respiratory activity is desirable in many clinical applications to detect respiratory events. Non-contact monitoring of respiration can be achieved with near- and far-infrared spectrum cameras. However, current technologies are not sufficiently robust to be used in clinical applications. For example, they fail to estimate an accurate respiratory rate (RR) during apnea. We present a novel algorithm based on multispectral data fusion that aims at estimating RR also during apnea. The algorithm independently addresses the RR estimation and apnea detection tasks. Respiratory information is extracted from multiple sources and fed into an RR estimator and an apnea detector whose results are fused into a final respiratory activity estimation. We evaluated the system retrospectively using data from 30 healthy adults who performed diverse controlled breathing tasks while lying supine in a dark room and reproduced central and obstructive apneic events. Combining multiple respiratory information from multispectral cameras improved the root mean square error (RMSE) accuracy of the RR estimation from up to 4.64 monospectral data down to 1.60 breaths/min. The median F1 scores for classifying obstructive (0.75 to 0.86) and central apnea (0.75 to 0.93) also improved. Furthermore, the independent consideration of apnea detection led to a more robust system (RMSE of 4.44 vs. 7.96 breaths/min). Our findings may represent a step towards the use of cameras for vital sign monitoring in medical applications.

Multi-source Information Fusion Technology and Its Engineering Application

With the continuous development of information technology in recent years, information fusion technology, which originated from military applications, plays an important role in various fields. In addition, the rapidly increasing amount of data and the changing lifestyles of people in the information age are affecting the development of information fusion technology. More experts and scholars have focused their attention on the research of image or audio and video fusion or distributed fusion technology. This article summarizes the origin and development of information fusion technology and typical algorithms, as well as the future development trends and challenges of information fusion technology.

Fusion of Video and Inertial Sensing for Deep Learning-Based Human Action Recognition.

This paper presents the simultaneous utilization of video images and inertial signals that are captured at the same time via a video camera and a wearable inertial sensor within a fusion framework in order to achieve a more robust human action recognition compared to the situations when each sensing modality is used individually. The data captured by these sensors are turned into 3D video images and 2D inertial images that are then fed as inputs into a 3D convolutional neural network and a 2D convolutional neural network, respectively, for recognizing actions. Two types of fusion are considered—Decision-level fusion and feature-level fusion. Experiments are conducted using the publicly available dataset UTD-MHAD in which simultaneous video images and inertial signals are captured for a total of 27 actions. The results obtained indicate that both the decision-level and feature-level fusion approaches generate higher recognition accuracies compared to the approaches when each sensing modality is used individually. The highest accuracy of 95.6% is obtained for the decision-level fusion approach.

Multi-scale Guided Image and Video Fusion: A Fast and Efficient Approach

In this paper, we propose a general purpose, simple and fast fusion algorithm based on guided image filter. The proposed method can well combine useful source image information into the fused image supported by multi-scale image decomposition, structure transferring property, visual saliency detection, and weight map construction. Multi-scale image decomposition is appropriate to represent and manipulate image features at various scales. Structure transferring property enabled by our algorithm can induce structures of one source image into the other. A new visual saliency detection based on guided image filter introduced in this paper is able to extract significant regions from visually different images of the same scene. The choice of weight maps helped to integrate the complementary information pixel by pixel at each scale. Experimental outcomes of the proposed method are compared and analyzed with traditional and recent guided image filter-based fusion algorithms in terms of visual quality, fusion metrics and run time. In addition, to enhance fusion results further we made an effort to find a suitable image and video enhancement algorithm. The fusion performance analysis clearly indicates that the proposed method is very promising along with less run time.

Rolling 3D Laplacian Pyramid Video Fusion

In this paper, we present a novel algorithm for video fusion of multi-sensor sequences applicable to real-time night vision systems. We employ the Laplacian pyramid fusion of a block of successive frames to add temporal robustness to the fused result. For the fusion rule, we first group high and low frequency levels of the decomposed frames in the block from both input sensor sequences. Then, we define local space-time energy measure to guide the selection based fusion process in a manner that achieves spatio-temporal stability. We demonstrate our approach on several well-known multi-sensor video fusion examples with varying contents and target appearance and show its advantage over conventional video fusion approaches. Computational complexity of the proposed methods is kept low by the use of simple linear filtering that can be easily parallelised for implementation on general-purpose graphics processing units (GPUs).