Video Fusion Research Articles

An infrared and visible light video fusion method based on chaos theory and PID control

Differences in the imaging mechanisms of infrared and visible light images lead to differences in the way their visually meaningful gradients are formed. Existing fusion methods use the same feature extractor to extract features from the source video frames, which ignores the differences in the gradients of video frames from different modalities. In this paper, we propose an infrared and visible light video fusion method based on chaos theory and proportional integral differential (PID) control. Firstly, for the Lorentz chaotic system, we give it initial values and parameters, and iterate it to obtain three scrambling sequences, through which the source video frames are scrambled in the rows, columns, and diagonal directions respectively to eliminate their visually meaningful gradients, so that the features extracted are of comparable scales in the same layer, and the fusion process can be carried out in the scale-consistent space. Second, we propose a structure-aware relative total variation feature extraction method (saRTV) for the two-scale decomposition of source video frames, which can transfer more features of source video frames to the detail layer. Then, based on our previous work, this paper introduces PID control to construct a closed-loop control system through transfer function design, controller design and measurement function design. This control system is used to fuse the detail layer, so as to realize the real-time guidance of the source video frames to the fusion process. Experiments on public datasets demonstrate that our method has better performance compared to some state-of-the-art methods.

Residual multimodal Transformer for expression‐EEG fusion continuous emotion recognition

AbstractContinuous emotion recognition is to predict emotion states through affective information and more focus on the continuous variation of emotion. Fusion of electroencephalography (EEG) and facial expressions videos has been used in this field, while there are with some limitations in current researches, such as hand‐engineered features, simple approaches to integration. Hence, a new continuous emotion recognition model is proposed based on the fusion of EEG and facial expressions videos named residual multimodal Transformer (RMMT). Firstly, the Resnet50 and temporal convolutional network (TCN) are utilised to extract spatiotemporal features from videos, and the TCN is also applied to process the computed EEG frequency power to acquire spatiotemporal features of EEG. Then, a multimodal Transformer is used to fuse the spatiotemporal features from the two modalities. Furthermore, a residual connection is introduced to fuse shallow features with deep features which is verified to be effective for continuous emotion recognition through experiments. Inspired by knowledge distillation, the authors incorporate feature‐level loss into the loss function to further enhance the network performance. Experimental results show that the RMMT reaches a superior performance over other methods for the MAHNOB‐HCI dataset. Ablation studies on the residual connection and loss function in the RMMT demonstrate that both of them is functional.

Surfacelet-IHS transform-based fusion of SAR and optical video

Surfacelet-IHS transform-based fusion of SAR and optical video

Multimodal fall detection for solitary individuals based on audio-video decision fusion processing

Falls often pose significant safety risks to solitary individuals, especially the elderly. Implementing a fast and efficient fall detection system is an effective strategy to address this hidden danger. We propose a multimodal method based on audio and video. On the basis of using non-intrusive equipment, it reduces to a certain extent the false negative situation that the most commonly used video-based methods may face due to insufficient lighting conditions, exceeding the monitoring range, etc. Therefore, in the foreseeable future, methods based on audio and video fusion are expected to become the best solution for fall detection. Specifically, this article outlines the following methodology: the video-based model utilizes YOLOv7-Pose to extract key skeleton joints, which are then fed into a two stream Spatial Temporal Graph Convolutional Network (ST-GCN) for classification. Meanwhile, the audio-based model employs log-scaled mel spectrograms to capture different features, which are processed through the MobileNetV2 architecture for detection. The final decision fusion of the two results is achieved through linear weighting and Dempster-Shafer (D-S) theory. After evaluation, our multimodal fall detection method significantly outperforms the single modality method, especially the evaluation metric sensitivity increased from 81.67% in single video modality to 96.67% (linear weighting) and 97.50% (D-S theory), which emphasizing the effectiveness of integrating video and audio data to achieve more powerful and reliable fall detection in complex and diverse daily life environments.

Enhanced Seamless Video Fusion: A Convolutional Pyramid-Based 3D Integration Algorithm.

Video fusion aims to synthesize video footage from different sources into a unified, coherent output. It plays a key role in areas such as video editing and special effects production. The challenge is to ensure the quality and naturalness of synthetic video, especially when dealing with footage of different sources and qualities. Researchers continue to strive to optimize algorithms to adapt to a variety of complex application scenarios and improve the effectiveness and applicability of video fusion. We introduce an algorithm based on a convolution pyramid and propose a 3D video fusion algorithm that looks for the potential function closest to the gradient field in the least square sense. The 3D Poisson equation is solved to realize seamless video editing. This algorithm uses a multi-scale method and wavelet transform to approximate linear time. Through numerical optimization, a small core is designed to deal with large target filters, and multi-scale transformation analysis and synthesis are realized. In terms of seamless video fusion, it shows better performance than existing algorithms. Compared with editing multiple 2D images into video after Poisson fusion, the video quality produced by this method is very close, and the computing speed of the video fusion is improved to a certain extent.

An Analysis on English Teachers’ Effective Classroom Discourse and Its Interactive Model Innovation with the Assistance of Artificial Intelligence

Abstract The rapid evolution of technology catalyzes innovation in English teaching methodologies, rendering traditional approaches—predominantly reliant on singular language modalities—insufficient for modern educational demands. This paper introduces a multimodal English teaching model anchored in multimodal discourse analysis and interaction theory to address these evolving needs. Utilizing a dual threshold extraction algorithm for chunked cumulative frame difference, the model effectively identifies semantic key frames during multimodal English instruction. Additionally, it incorporates a composite convolutional neural network that merges spatio-temporal features to delineate the dynamics of the teaching videos, further enhanced by the LK optical flow method. The integration of a pre-interactive LSTM decoder facilitates the fusion of video and textual features, culminating in the construction of an annotated English teaching video model. This model was applied to analyze the classroom discourse of high school sophomores’ English teachers in X city. Analysis revealed that the duration of classroom discourse varied between 14 to 19 minutes per session, with the introductory segment alone accounting for 307.54 seconds—12.81% of the total lesson duration—of purely linguistic content. Furthermore, the average number of questions posed by teachers per lesson was 73.17, indicating a high reliance on verifying discourse strategies, which constituted 92.22% of the discourse. This technological approach to analyzing classroom discourse provides novel insights and valuable references for refining English teaching strategies, demonstrating the effectiveness of integrating advanced computational techniques in educational settings.

Non-contact human respiratory rate measurement based on two-level fusions of video and FMCW radar information

In this paper, we propose a robust respiratory rate (RR) measurement method using a two-level fusion of video and FMCW (frequency modulated continuous wave) radar information. Specifically, the video pixel displacement signals and the radar phase variation signals are extracted independently in shoulders and chest regions, which are then denoised by the serial-CEEMDAN (SCEEMDAN) method to obtain the corresponding single modal RR values. Next, at the feature level, the multivariate singular spectrum analysis (MSSA) is employed to extract the shared respiratory components in the video and radar modalities. Finally, at the decision level, we calculate the target RR value based on a signal-to-noise ratio (SNR) weighting of all single-modality and MSSA results. The proposed two-level fusion method is evaluated on a self-collected dataset, which includes 15 healthy subjects acting five challenging activities. The experimental results show that the proposed fusion method significantly outperform that of the single modality.

Attention-Based Fusion of Ultrashort Voice Utterances and Depth Videos for Multimodal Person Identification

Multimodal deep learning, in the context of biometrics, encounters significant challenges due to the dependence on long speech utterances and RGB images, which are often impractical in certain situations. This paper presents a novel solution addressing these issues by leveraging ultrashort voice utterances and depth videos of the lip for person identification. The proposed method utilizes an amalgamation of residual neural networks to encode depth videos and a Time Delay Neural Network architecture to encode voice signals. In an effort to fuse information from these different modalities, we integrate self-attention and engineer a noise-resistant model that effectively manages diverse types of noise. Through rigorous testing on a benchmark dataset, our approach exhibits superior performance over existing methods, resulting in an average improvement of 10%. This method is notably efficient for scenarios where extended utterances and RGB images are unfeasible or unattainable. Furthermore, its potential extends to various multimodal applications beyond just person identification.

Detail-aware near infrared and visible fusion with multi-order hyper-Laplacian priors

Heavy haze/noise can cause unpleasant information loss in near infrared (NIR) and visible (VI) image fusion. To generate high-quality fused images, this paper proposes a detail-aware near infrared and visible image fusion method, which establishes a detail injection variational framework based on multi-order hyper-Laplacian priors (MHLP). As the core idea, the MHLP is a l1/2 norm penalty term with first-order and second-order gradient differences, and applies sparsity-promoting and complete-comprehensive injection terms to enhance salient structures and fine-scale details. Besides, a noise map is introduced to suppress the structure inconsistency caused by heavy noise. Combining MHLP and a noise map, a novel near infrared and visible fusion framework is organized to deal with harsh environment imaging problems. The framework recovers visual visibility and preserves the significant scenery structural information and intrinsic colors. The video fusion of near infrared and visible also can be realized by the framework, which introduces the inter-frame coherence of successive frames additionally to retain the information lost in the degraded scene. Experiment results on public datasets and industrial sites demonstrate the advantages of our method from both qualitative and quantitative perspectives. The code is available at: https://github.com/BOYang-pro/MHLP.

Reproduction of spatial–temporal distribution of traffic loads on freeway bridges via fusion of camera video and ETC data

Understanding the spatial–temporal distribution of traffic loads on bridge decks is essential for managing traffic flow and assessing bridge load effects. While computer vision can already identify on-bridge vehicle parameters through traffic surveillance cameras, obtaining cost-effective vehicle load information remains a challenge, even though it is the most important service load for bridges. In this study, we present a methodology that applies computer vision algorithms to fuzzy match video-identified data with electronic toll collection (ETC) recorded data to reproduce the spatial–temporal distribution of traffic loads on freeway bridges. The proposed method involves three critical steps. First, multi-object detection and trajectory tracking of vehicle contours, vehicle heads, vehicle tires, and vehicle license plates are implemented using YOLOv5 and DeepSort algorithms. Second, the conversion of image coordinates to real-world coordinates is automatically calibrated using pre-measured roadside reflection points. Third, vehicle license plate characters are identified by the LPRnet algorithm and the results are fused with highway ETC records to determine vehicle weight information, while the KNN clustering algorithm is used to further infer vehicle axle weight. After these three steps, the vehicle load sequence in the video surveillance area can be reproduced. Finally, the effectiveness and feasibility of the proposed method are verified in an example of a highway bridge.

Fusion of video and inertial sensing data via dynamic optimization of a biomechanical model

Inertial sensing and computer vision are promising alternatives to traditional optical motion tracking, but until now these data sources have been explored either in isolation or fused via unconstrained optimization, which may not take full advantage of their complementary strengths. By adding physiological plausibility and dynamical robustness to a proposed solution, biomechanical modeling may enable better fusion than unconstrained optimization. To test this hypothesis, we fused video and inertial sensing data via dynamic optimization with a nine degree-of-freedom model and investigated when this approach outperforms video-only, inertial-sensing-only, and unconstrained-fusion methods. We used both experimental and synthetic data that mimicked different ranges of video and inertial measurement unit (IMU) data noise. Fusion with a dynamically constrained model significantly improved estimation of lower-extremity kinematics over the video-only approach and estimation of joint centers over the IMU-only approach. It consistently outperformed single-modality approaches across different noise profiles. When the quality of video data was high and that of inertial data was low, dynamically constrained fusion improved estimation of joint kinematics and joint centers over unconstrained fusion, while unconstrained fusion was advantageous in the opposite scenario. These findings indicate that complementary modalities and techniques can improve motion tracking by clinically meaningful margins and that data quality and computational complexity must be considered when selecting the most appropriate method for a particular application.

Web-based mixed reality video fusion with remote rendering

BackgroundMixed reality (MR) video fusion systems merge video imagery with 3D scenes to make the scene more realistic and help users understand the video content and temporal–spatial correlation between them, reducing the user′s cognitive load. MR video fusion are used in various applications; however, video fusion systems require powerful client machines because video streaming delivery, stitching, and rendering are computationally intensive. Moreover, huge bandwidth usage is another critical factor that affects the scalability of video-fusion systems. MethodsOur framework proposes a fusion method for dynamically projecting video images into 3D models as textures. ResultsSeveral experiments on different metrics demonstrate the effectiveness of the proposed framework. ConclusionsThe framework proposed in this study can overcome client limitations by utilizing remote rendering. Furthermore, the framework we built is based on browsers. Therefore, the user can test the MR video fusion system with a laptop or tablet without installing any additional plug-ins or application programs.

Dynamic Fusion Technology of Mobile Video and 3D GIS: The Example of Smartphone Video

Mobile videos contain a large amount of data, where the information interesting to the user can either be discrete or distributed. This paper introduces a method for fusing 3D geographic information systems (GIS) and video image textures. For the dynamic fusion of video in 3DGIS where the position and pose angle of the filming device change moment by moment, it integrates GIS 3D visualization, pose resolution and motion interpolation, and proposes a projection texture mapping method for constructing a dynamic depth camera to achieve dynamic fusion. In this paper, the accuracy and time efficiency of different systems of gradient descent and complementary filtering algorithms are analyzed mainly by quantitative analysis method, and the effect of dynamic fusion is analyzed by the playback delay and rendering frame rate of video on 3DGIS as indicators. The experimental results show that the gradient descent method under the Aerial Attitude Reference System (AHRS) is more suitable for the solution of smartphone attitude, and can control the root mean square error of attitude solution within 2°; the delay of video playback on 3DGIS is within 29 ms, and the rendering frame rate is 34.9 fps, which meets the requirements of the minimum resolution of human eyes.

Fusion of CCTV Video and Spatial Information for Automated Crowd Congestion Monitoring in Public Urban Spaces

Crowd congestion is one of the main causes of modern public safety issues such as stampedes. Conventional crowd congestion monitoring using closed-circuit television (CCTV) video surveillance relies on manual observation, which is tedious and often error-prone in public urban spaces where crowds are dense, and occlusions are prominent. With the aim of managing crowded spaces safely, this study proposes a framework that combines spatial and temporal information to automatically map the trajectories of individual occupants, as well as to assist in real-time congestion monitoring and prediction. Through exploiting both features from CCTV footage and spatial information of the public space, the framework fuses raw CCTV video and floor plan information to create visual aids for crowd monitoring, as well as a sequence of crowd mobility graphs (CMGraphs) to store spatiotemporal features. This framework uses deep learning-based computer vision models, geometric transformations, and Kalman filter-based tracking algorithms to automate the retrieval of crowd congestion data, specifically the spatiotemporal distribution of individuals and the overall crowd flow. The resulting collective crowd movement data is then stored in the CMGraphs, which are designed to facilitate congestion forecasting at key exit/entry regions. We demonstrate our framework on two video data, one public from a train station dataset and the other recorded at a stadium following a crowded football game. Using both qualitative and quantitative insights from the experiments, we demonstrate that the suggested framework can be useful to help assist urban planners and infrastructure operators with the management of congestion hazards.

Visualization of Environmental Sensing Data in the Lake-Oriented Digital Twin World: Poyang Lake as an Example

Access to real-time environmental sensing data is key to representing real-time environmental changes in the digital twin of lakes. The visualization of environmental sensing data is an important part of establishing a digital twin of lakes. In the past, environmental sensing data display methods were either charts or two-dimensional map-based visualization methods. Breaking through the traditional visualization methods of environmental sensing data and realizing a multi-dimensional and multi-view display of environmental sensing data in a digital twin of lakes is something that this particular paper tries to resolve. This study proposes a visualization framework to integrate, manage, analyze, and visualize the environmental sensing data in the digital twin of lakes. In addition, this study also seeks to realize the coupling expression of geospatial data and long-term monitoring sequence data. Different visualization methods are used to realize the visualization of environmental sensing data in the digital twin of lakes. Using vector and scalar visualization methods to display ambient wireless sensor monitoring data in a digital twin of lakes provides researchers with richer visualization methods and means for deeper analysis. Using video fusion technology to display environmental sensing video surveillance data strengthens the integration of the virtual environment and real space and saves time for position identification using video surveillance. These findings may also help realize the integration and management of real-time environmental sensing data in a digital twin of lakes. The visualization framework uses various visualization methods to express the monitoring data of environmental wireless sensors and increases the means of visualizing environmental sensing data in the world of a lake digital twin. This visualization framework is also a general approach that can be applied to all similar lakes, or other geographical scenarios where environmental sensing devices are deployed. The establishment of a digital twin of Poyang Lake has certain practical significance for improving the digital management level of Poyang Lake and monitoring its ecological changes. Poyang Lake is used as an example to verify the proposed framework and method, which shows that the framework can be applied to the construction of a lake-oriented digital twin.

Real‐time fusion of multiple videos and 3D real scenes based on optimal viewpoint selection

Abstract3D real scenes are a digital virtual space that represents and portrays the real world in a photorealistic, three‐dimensional, and sequential manner. Existing methods for constructing and updating 3D models, such as oblique photography and laser scanning, are difficult to meet the demand of perceiving the real world intuitively, dynamically, and in real time. In recent years, the method of integrating rapidly rising video data and 3D models has become increasingly popular. Compared with existing methods, it enhances the real‐time perception of 3D scenes by taking advantage of the real‐time character of videos and the intuitive character of 3D models. In this article, we propose a real‐time fusion method of multiple videos and 3D real scenes based on optimal viewpoint selection. To begin, 3D reconstruction and video camera calibration were used to prepare the basic data for the fusion of videos and 3D model. Second, a visible‐surface detection‐based video space restoration method was provided, and the overlapping region between multiple videos was determined. Third, to split the overlapping region into the corresponding camera spaces, a segmentation method based on optimal viewpoint selection was given. Finally, the 2D videos were dynamically fitted to the 3D model using the dynamic texture mapping method, while accomplishing the fusing and rendering of the 3D real scene. After experimental verification, the overall effect of the multiple videos and 3D real scene fusion system implemented using the method proposed in this article is better, while the algorithm is less time‐consuming and efficient in rendering.

An Accurate Urine Red Blood Cell Detection Method Based on Multi-Focus Video Fusion and Deep Learning with Application to Diabetic Nephropathy Diagnosis

Background and Objective: Detecting urine red blood cells (U-RBCs) is an important operation in diagnosing nephropathy. Existing U-RBC detection methods usually employ single-focus images to implement such tasks, which inevitably results in false positives and missed detections due to the abundance of defocused U-RBCs in the single-focus images. Meanwhile, the current diabetic nephropathy diagnosis methods heavily rely on artificially setting a threshold to detect the U-RBC proportion, whose accuracy and robustness are still supposed to be improved. Methods: To overcome these limitations, a novel multi-focus video dataset in which the typical shape of all U-RBCs can be captured in one frame is constructed, and an accurate U-RBC detection method based on multi-focus video fusion (D-MVF) is presented. The proposed D-MVF method consists of multi-focus video fusion and detection stages. In the fusion stage, D-MVF first uses the frame-difference data of multi-focus video to separate the U-RBCs from the background. Then, a new key frame extraction method based on the three metrics of information entropy, edge gradient, and intensity contrast is proposed. This method is responsible for extracting the typical shapes of U-RBCs and fusing them into a single image. In the detection stage, D-MVF utilizes the high-performance deep learning model YOLOv4 to rapidly and accurately detect U-RBCs based on the fused image. In addition, based on U-RBC detection results from D-MVF, this paper applies the K-nearest neighbor (KNN) method to replace artificial threshold setting for achieving more accurate diabetic nephropathy diagnosis. Results: A series of controlled experiments are conducted on the self-constructed dataset containing 887 multi-focus videos, and the experimental results show that the proposed D-MVF obtains a satisfactory mean average precision (mAP) of 0.915, which is significantly higher than that of the existing method based on single-focus images (0.700). Meanwhile, the diabetic nephropathy diagnosis accuracy and specificity of KNN reach 0.781 and 0.793, respectively, which significantly exceed the traditional threshold method (0.719 and 0.759). Conclusions: The research in this paper intelligently assists microscopists to complete U-RBC detection and diabetic nephropathy diagnosis. Therefore, the work load of microscopists can be effectively relieved, and the urine test demands of nephrotic patients can be met.

Night Vision Anti-Halation Method Based on Infrared and Visible Video Fusion.

In order to address the discontinuity caused by the direct application of the infrared and visible image fusion anti-halation method to a video, an efficient night vision anti-halation method based on video fusion is proposed. The designed frame selection based on inter-frame difference determines the optimal cosine angle threshold by analyzing the relation of cosine angle threshold with nonlinear correlation information entropy and de-frame rate. The proposed time-mark-based adaptive motion compensation constructs the same number of interpolation frames as the redundant frames by taking the retained frame number as a time stamp. At the same time, considering the motion vector of two adjacent retained frames as the benchmark, the adaptive weights are constructed according to the interframe differences between the interpolated frame and the last retained frame, then the motion vector of the interpolated frame is estimated. The experimental results show that the proposed frame selection strategy ensures the maximum safe frame removal under the premise of continuous video content at different vehicle speeds in various halation scenes. The frame numbers and playing duration of the fused video are consistent with that of the original video, and the content of the interpolated frame is highly synchronized with that of the corresponding original frames. The average FPS of video fusion in this work is about six times that in the frame-by-frame fusion, which effectively improves the anti-halation processing efficiency of video fusion.

MLF: A mimic layered fusion method for infrared and visible video

Optimal selection of fusion strategy based on image difference information is an important way to improve the adaptive performance of infrared photoelectric detection systems. Aiming at the problem that the current infrared and visible video fusion model cannot be adjusted dynamically according to the difference features of video intra-frame, resulting in poor fusion effect or even fusion failure. In this paper, a mimic layered fusion method for infrared and visible video is proposed. Firstly, by comparing the general video fusion process and polymorphic process of the mimic octopus, we establish the corresponding relationship of them, and determine the four-layer variants of video mimic layered fusion. Secondly, we divide the region of interest in each frame of the video sequence, frames with significant changes in the region of interest are selected as salient frames. Thirdly, the magnitudes of various difference features of the region of interest of the salient frames are calculated respectively. Then, and a fusion effectiveness function is constructed based on cosine similarity and weighting idea to analyze and compare the fusion effects of fusion algorithms, fusion rules, fusion parameters and fusion structures on various difference features layer by layer, so as to select the optimal mimetic variant layer by layer, the mimic layered fusion is realized based on layered fusion discrimination mechanism through the optimal combination of variants. Finally, the experimental results show that our method in this paper has achieved remarkable results in preserving the typical infrared target and visible structural details in the whole video, and is significantly better than other single fusion methods in quantitative analysis and qualitative evaluation.

Joint modelling of audio-visual cues using attention mechanisms for emotion recognition

Emotions play a crucial role in human-human communications with complex socio-psychological nature. In order to enhance emotion communication in human-computer interaction, this paper studies emotion recognition from audio and visual signals in video clips, utilizing facial expressions and vocal utterances. Thereby, the study aims to exploit temporal information of audio-visual cues and detect their informative time segments. Attention mechanisms are used to exploit the importance of each modality over time. We propose a novel framework that consists of bi-modal time windows spanning short video clips labeled with discrete emotions. The framework employs two networks, with each one being dedicated to one modality. As input to a modality-specific network, we consider a time-dependent signal deriving from the embeddings of the video and audio modalities. We employ the encoder part of the Transformer on the visual embeddings and another one on the audio embeddings. The research in this paper introduces detailed studies and meta-analysis findings, linking the outputs of our proposition to research from psychology. Specifically, it presents a framework to understand underlying principles of emotion recognition as functions of three separate setups in terms of modalities: audio only, video only, and the fusion of audio and video. Experimental results on two datasets show that the proposed framework achieves improved accuracy in emotion recognition, compared to state-of-the-art techniques and baseline methods not using attention mechanisms. The proposed method improves the results over baseline methods by at least 5.4%. Our experiments show that attention mechanisms reduce the gap between the entropies of unimodal predictions, which increases the bimodal predictions’ certainty and, therefore, improves the bimodal recognition rates. Furthermore, evaluations with noisy data in different scenarios are presented during the training and testing processes to check the framework’s consistency and the attention mechanism’s behavior. The results demonstrate that attention mechanisms increase the framework’s robustness when exposed to similar conditions during the training and the testing phases. Finally, we present comprehensive evaluations of emotion recognition as a function of time. The study shows that the middle time segments of a video clip are essential in the case of using audio modality. However, in the case of video modality, the importance of time windows is distributed equally.