Fusion Architecture Research Articles

Self-adaptive spatial-temporal network based on heterogeneous data for air quality prediction

With the development of society and the rise of people's environmental awareness, air pollution is receiving increased public attention. Accurate air quality prediction can provide useful information for government decision-making and residents' activities. However, accurately predicting future air quality remains a challenging task because of the complex spatial-temporal dependencies of air quality. Previous studies failed to explicitly model these spatial-temporal dependencies. In this paper, we propose a self-adaptive spatial-temporal network (SA-STNet) to efficiently and effectively capture the spatial-temporal dependencies of air quality. In order to effectively aggregate spatial information, we employ a self-adaptive graph convolution module that can learn the latent spatial correlations of air quality automatically. In the temporal dimension, we utilise three independent components to capture the recent, daily-periodic, and weekly-periodic temporal dependencies of air quality, respectively. In addition, our model exploits rich external complementary information by means of a features extraction component. A parametric-matrix-based fusion architecture is used to combine the outputs of different components into a joint representation which is used for generating the final prediction results. Extensive experiments carried out on real-world datasets demonstrate the outstanding performance of our model compared with baselines and state-of-the-art methods.

Jean Renaudie en Vaudreuil (1967-1968): grafismo abstracto y morfología paisajística de un urbanismo utópico

<p>Este artículo examina los dibujos de Jean Renaudie para la nueva ciudad de Vaudreuil (1967-1968). El análisis de su estructura, su cromatismo y su formalidad, situará aquel conjunto de croquis como el punto de inflexión hacia la fusión de arquitectura y urbanismo en un marco teórico único. El desarrollo abordará el propósito de Renaudie de expresar la complejidad y el deseo de cohesión social, en un grafismo dispuesto a exponer su idea de “la ville est une combinatoire” frente al zonning funcionalista de la Carta de Atenas, fuertemente criticado por su generación. El objetivo será demostrar si aquella abstracción gráfica sugería ese “espacio absolutamente otro”, capaz de alojar la nueva sociedad por la que abogaba la teoría social de sus referentes intelectuales -Althusser, Foucault, Lévy Strauss, Lefebvre, Marcuse-. Y si, a su vez, supuso el germen de materializaciones posteriores como las desarrolladas en los conjuntos urbanos de Ivry-sur-Seine, Givors o Saint Martin d’Hères.<em></em></p>

Structural similarity preserving GAN for infrared and visible image fusion

Compared with a single image, in a complex environment, image fusion can utilize the complementary information provided by multiple sensors to significantly improve the image clarity and the information, more accurate, reliable, comprehensive access to target and scene information. It is widely used in military and civil fields, such as remote sensing, medicine, security and other fields. In this paper, we propose an end-to-end fusion framework based on structural similarity preserving GAN (SSP-GAN) to learn a mapping of the fusion tasks for visible and infrared images. Specifically, on the one hand, for making the fusion image natural and conforming to visual habits, structure similarity is introduced to guide the generator network produce abundant texture structure information. On the other hand, to fully take advantage of shallow detail information and deep semantic information for achieving feature reuse, we redesign the network architecture of multi-modal image fusion meticulously. Finally, a wide range of experiments on real infrared and visible TNO dataset and RoadScene dataset prove the superior performance of the proposed approach in terms of accuracy and visual. In particular, compared with the best results of other seven algorithms, our model has improved entropy, edge information transfer factor, multi-scale structural similarity and other evaluation metrics, respectively, by 3.05%, 2.4% and 0.7% on TNO dataset. And our model has also improved by 0.7%, 2.82% and 1.1% on RoadScene dataset.

A hierarchical fusion framework to integrate homogeneous and heterogeneous classifiers for medical decision-making

Classifier diversity and fusion architecture are two critical characteristics stressed in homogeneous and heterogeneous ensemble learning methods and they are equally important for building a successful multi-classifier system. In this study, we introduced a two-level framework, namely hierarchical fusion of homogeneous and heterogeneous multi-classifiers (HF2HM), to integrate the diversified classification models produced by feeding heterogeneous classifiers with homogeneous random-projected training datasets. The proposed hierarchical fusion scheme was comprehensively validated using fifteen public UCI datasets and three clinical datasets. The experimental results demonstrated the superiority of the proposed HF2HM framework over the base classifiers and the state-of-the-art benchmark ensemble methods, verifying it as a potential tool to assist in medical decision making in practical clinical settings.

Deep Hashing for Secure Multimodal Biometrics

When compared to unimodal systems, multimodal biometric systems have several advantages, including lower error rate, higher accuracy, and larger population coverage. However, multimodal systems have an increased demand for integrity and privacy because they must store multiple biometric traits associated with each user. In this paper, we present a deep learning framework for feature-level fusion that generates a secure multimodal template from each user's face and iris biometrics. We integrate a deep hashing (binarization) technique into the fusion architecture to generate a robust binary multimodal shared latent representation. Further, we employ a hybrid secure architecture by combining cancelable biometrics with secure sketch techniques and integrate it with a deep hashing framework, which makes it computationally prohibitive to forge a combination of multiple biometrics that pass the authentication. The efficacy of the proposed approach is shown using a multimodal database of face and iris and it is observed that the matching performance is improved due to the fusion of multiple biometrics. Furthermore, the proposed approach also provides cancelability and unlinkability of the templates along with improved privacy of the biometric data. Additionally, we also test the proposed hashing function for an image retrieval application using a benchmark dataset. The main goal of this paper is to develop a method for integrating multimodal fusion, deep hashing, and biometric security, with an emphasis on structural data from modalities like face and iris. The proposed approach is in no way a general biometric security framework that can be applied to all biometric modalities, as further research is needed to extend the proposed framework to other unconstrained biometric modalities.

Research on Urban Traffic Signal Control Systems Based on Cyber Physical Systems

In this paper, we present a traffic cyber physical system for urban road traffic signal control, which is referred to as UTSC-CPS. With this proposed system, managers and researchers can realize the construction and simulation of various types of traffic scenarios, the rapid development, and optimization of new control strategies and can apply effective control strategies to actual traffic management. The advantages of this new system include the following. Firstly, the fusion architecture of private cloud computing and edge computing is proposed for the first time, which effectively improves the performance of software and hardware of the urban road traffic signal control system and realizes information security perception and protection in cloud and equipment, respectively, within the fusion framework; secondly, using the concept of parallel system, the depth of real-time traffic control subsystem and real-time simulation subsystem is realized. Thirdly, the idea of virtual scene basic engine and strategy agent engine is put forward in the system design, which separates data from control strategy by designing a general control strategy API and helps researchers focus on control algorithm itself without paying attention to detection data and basic data. Finally, considering China, the system designs a general control strategy API to separate data from control strategy. Most of the popular communication protocols between signal controllers and detectors are private protocols. The standard protocol conversion middleware is skillfully designed, which decouples the field equipment from the system software and achieves the universality and reliability of the control strategy. To further demonstrate the advantages of the new system, we have carried out a one-year practical test in Weifang City, Shandong Province, China. The system has been proved in terms of stability, security, scalability, practicability and rapid practice, and verification of the new control strategy. At the same time, it proves the superiority of the simulation subsystem in the performance and simulation scale by comparing the different-scale road networks of Shunyi District in Beijing and Weifang City in Shandong Province. Further tests were conducted using real intersections, and the results were equally valid.

Deep Dual-Modal Traffic Objects Instance Segmentation Method Using Camera and LIDAR Data for Autonomous Driving

Recent advancements in environmental perception for autonomous vehicles have been driven by deep learning-based approaches. However, effective traffic target detection in complex environments remains a challenging task. This paper presents a novel dual-modal instance segmentation deep neural network (DM-ISDNN) by merging camera and LIDAR data, which can be used to deal with the problem of target detection in complex environments efficiently based on multi-sensor data fusion. Due to the sparseness of the LIDAR point cloud data, we propose a weight assignment function that assigns different weight coefficients to different feature pyramid convolutional layers for the LIDAR sub-network. We compare and analyze the adaptations of early-, middle-, and late-stage fusion architectures in depth. By comprehensively considering the detection accuracy and detection speed, the middle-stage fusion architecture with a weight assignment mechanism, with the best performance, is selected. This work has great significance for exploring the best feature fusion scheme of a multi-modal neural network. In addition, we apply a mask distribution function to improve the quality of the predicted mask. A dual-modal traffic object instance segmentation dataset is established using a 7481 camera and LIDAR data pairs from the KITTI dataset, with 79,118 manually annotated instance masks. To the best of our knowledge, there is no existing instance annotation for the KITTI dataset with such quality and volume. A novel dual-modal dataset, composed of 14,652 camera and LIDAR data pairs, is collected using our own developed autonomous vehicle under different environmental conditions in real driving scenarios, for which a total of 62,579 instance masks are obtained using semi-automatic annotation method. This dataset can be used to validate the detection performance under complex environmental conditions of instance segmentation networks. Experimental results on the dual-modal KITTI Benchmark demonstrate that DM-ISDNN using middle-stage data fusion and the weight assignment mechanism has better detection performance than single- and dual-modal networks with other data fusion strategies, which validates the robustness and effectiveness of the proposed method. Meanwhile, compared to the state-of-the-art instance segmentation networks, our method shows much better detection performance, in terms of AP and F1 score, on the dual-modal dataset collected under complex environmental conditions, which further validates the superiority of our method.

Congestion-aware WiFi offload algorithm for 5G heterogeneous wireless networks

First, the key technologies of 5G heterogeneous wireless networks are introduced. Based on the 5G heterogeneous network fusion architecture, heterogeneous network technologies are used to achieve the integration between different access networks, so that they can cooperate with each other, coordinate interference, and improve spectrum resources. Utilization and transmission efficiency; At the same time, an enhanced ADNSF congestion-aware WiFi(WIreless FIdelity) offloading system suitable for 5G networks is proposed. The congestion-aware WiFi offloading system can obtain network status information in real time. Analyzed the shortcomings of 3GPP’s existing congestion-aware WiFi offload strategy, combined with the user-centric network characteristics of 5G, proposed a congestion-aware WiFi offload algorithm, and modeled the access strategy as the effective capacity In the non-cooperative game model, a distributed algorithm and a dynamic value algorithm are designed respectively to maximize network utility. Simulation shows that the proposed distributed algorithm doubles the total effective capacity of the network compared to the device full activation scheme; compared with the distributed algorithm, the proposed dynamic value algorithm improves the total effective capacity of the network. This algorithm can effectively improve user satisfaction and achieve fine management of congested WiFi offloads

Detection and segmentation of morphologically complex eukaryotic cells in fluorescence microscopy images via feature pyramid fusion.

Detection and segmentation of macrophage cells in fluorescence microscopy images is a challenging problem, mainly due to crowded cells, variation in shapes, and morphological complexity. We present a new deep learning approach for cell detection and segmentation that incorporates previously learned nucleus features. A novel fusion of feature pyramids for nucleus detection and segmentation with feature pyramids for cell detection and segmentation is used to improve performance on a microscopic image dataset created by us and provided for public use, containing both nucleus and cell signals. Our experimental results indicate that cell detection and segmentation performance significantly benefit from the fusion of previously learned nucleus features. The proposed feature pyramid fusion architecture clearly outperforms a state-of-the-art Mask R-CNN approach for cell detection and segmentation with relative mean average precision improvements of up to 23.88% and 23.17%, respectively.

More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification

Classification and identification of the materials lying over or beneath the Earth's surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS) and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the aforementioned difficulty by developing a general multimodal deep learning (MDL) framework. In particular, we also investigate a special case of multi-modality learning (MML) -- cross-modality learning (CML) that exists widely in RS image classification applications. By focusing on "what", "where", and "how" to fuse, we show different fusion strategies as well as how to train deep networks and build the network architecture. Specifically, five fusion architectures are introduced and developed, further being unified in our MDL framework. More significantly, our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs). To validate the effectiveness and superiority of the MDL framework, extensive experiments related to the settings of MML and CML are conducted on two different multimodal RS datasets. Furthermore, the codes and datasets will be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing to the RS community.

Deep Learning Based Multi-Modal Fusion Architectures for Maritime Vessel Detection

Object detection is a fundamental computer vision task for many real-world applications. In the maritime environment, this task is challenging due to varying light, view distances, weather conditions, and sea waves. In addition, light reflection, camera motion and illumination changes may cause to false detections. To address this challenge, we present three fusion architectures to fuse two imaging modalities: visible and infrared. These architectures can provide complementary information from two modalities in different levels: pixel-level, feature-level, and decision-level. They employed deep learning for performing fusion and detection. We investigate the performance of the proposed architectures conducting a real marine image dataset, which is captured by color and infrared cameras on-board a vessel in the Finnish archipelago. The cameras are employed for developing autonomous ships, and collect data in a range of operation and climatic conditions. Experiments show that feature-level fusion architecture outperforms the state-of-the-art other fusion level architectures.

U-GAN Model for Infrared and Visible Images Fusion

Infrared and visible image fusion is an effective method to solve the lack of single sensor imaging. The purpose is that the fusion images are suitable for human eyes and conducive to the next application and processing. In order to solve the problems of incomplete feature extraction, loss of details, and less samples of common data sets, it is not conducive to training, an end-to-end network architecture for image fusion is proposed. U-net is introduced into image fusion, and the final fusion result is obtained by using the generative adversarial network. Through its special convolution structure, the important feature information is extracted to the maximum extent, and the sample does not need to be cut to avoid the problem of reducing the fusion accuracy, but also to improve the training speed. Then the U-net extracted feature is confronted with the discriminator containing infrared image, and the generator model is obtained. The experimental results show that the present algorithm can obtain the fusion image with clear outline, prominent texture and obvious target. SD, SF, SSIM, AG and other indicators are obviously improved.

A Deep Learning Model for Fault Diagnosis with a Deep Neural Network and Feature Fusion on Multi-Channel Sensory Signals.

Collecting multi-channel sensory signals is a feasible way to enhance performance in the diagnosis of mechanical equipment. In this article, a deep learning method combined with feature fusion on multi-channel sensory signals is proposed. First, a deep neural network (DNN) made up of auto-encoders is adopted to adaptively learn representative features from sensory signal and approximate non-linear relation between symptoms and fault modes. Then, Locality Preserving Projection (LPP) is utilized in the fusion of features extracted from multi-channel sensory signals. Finally, a novel diagnostic model based on multiple DNNs (MDNNs) and softmax is constructed with the input of fused deep features. The proposed method is verified in intelligent failure recognition for automobile final drive to evaluate its performance. A set of contrastive analyses of several intelligent models based on the Back-Propagation Neural Network (BPNN), Support Vector Machine (SVM) and the proposed deep architecture with single sensory signal and multi-channel sensory signals is implemented. The proposed deep architecture of feature extraction and feature fusion on multi-channel sensory signals can effectively recognize the fault patterns of final drive with the best diagnostic accuracy of 95.84%. The results confirm that the proposed method is more robust and effective than other comparative methods in the contrastive experiments.

Leveraging Multimodal Deep Learning Architecture with Retina Lesion Information to Detect Diabetic Retinopathy.

PurposeTo improve disease severity classification from fundus images using a hybrid architecture with symptom awareness for diabetic retinopathy (DR).MethodsWe used 26,699 fundus images of 17,834 diabetic patients from three Taiwanese hospitals collected in 2007 to 2018 for DR severity classification. Thirty-seven ophthalmologists verified the images using lesion annotation and severity classification as the ground truth. Two deep learning fusion architectures were proposed: late fusion, which combines lesion and severity classification models in parallel using a postprocessing procedure, and two-stage early fusion, which combines lesion detection and classification models sequentially and mimics the decision-making process of ophthalmologists. Messidor-2 was used with 1748 images to evaluate and benchmark the performance of the architecture. The primary evaluation metrics were classification accuracy, weighted κ statistic, and area under the receiver operating characteristic curve (AUC).ResultsFor hospital data, a hybrid architecture achieved a good detection rate, with accuracy and weighted κ of 84.29% and 84.01%, respectively, for five-class DR grading. It also classified the images of early stage DR more accurately than conventional algorithms. The Messidor-2 model achieved an AUC of 97.09% in referral DR detection compared to AUC of 85% to 99% for state-of-the-art algorithms that learned from a larger database.ConclusionsOur hybrid architectures strengthened and extracted characteristics from DR images, while improving the performance of DR grading, thereby increasing the robustness and confidence of the architectures for general use.Translational RelevanceThe proposed fusion architectures can enable faster and more accurate diagnosis of various DR pathologies than that obtained in current manual clinical practice.

5G Nonstandalone Fusion Networks Based on the Intelligent Transplant Immune Tolerance

Fifth-generation (5G) nonstandalone (NSA) uses the 4G infrastructure to realize the rapid deployment of 5G, which will challenge the convergence of network architecture, signaling recognition, and editing capabilities of the next generation of mobile communications systems. Inspired by the immune technology of allograft transplantation, this article proposes for the first time the mechanism of antirejection induced by immune tolerance (IT) to solve the problem of signaling recognition and fusion between the heterogeneous mobile communication systems. First, with transplantation immunity as the technical background, a 5G NSA fusion architecture based on the adaptive IT mechanism was constructed. Then, the deep learning (DL) algorithm is used to efficiently identify the type of donor protocol, and the affinity calculation between the recipient and the donor is used to further enhance the correctness of the model recognition results. Meanwhile, the antibody concentration calculation was used to maintain the diversity of antibodies, which realized the complementarity between the IT mechanism and the DL algorithm. Finally, the donor signaling is decoded based on xor and encoded again based on the gene bit field. Simulation results show that the proposed method can effectively identify donor signaling, improve the affinity between the heterogeneous signaling and the efficiency of encoding and decoding, and broaden the research boundary of artificial immunity.

Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors

Gesture recognition using machine-learning methods is valuable in the development of advanced cybernetics, robotics and healthcare systems, and typically relies on images or videos. To improve recognition accuracy, such visual data can be combined with data from other sensors, but this approach, which is termed data fusion, is limited by the quality of the sensor data and the incompatibility of the datasets. Here, we report a bioinspired data fusion architecture that can perform human gesture recognition by integrating visual data with somatosensory data from skin-like stretchable strain sensors made from single-walled carbon nanotubes. The learning architecture uses a convolutional neural network for visual processing and then implements a sparse neural network for sensor data fusion and recognition at the feature level. Our approach can achieve a recognition accuracy of 100% and maintain recognition accuracy in non-ideal conditions where images are noisy and under- or over-exposed. We also show that our architecture can be used for robot navigation via hand gestures, with an error of 1.7% under normal illumination and 3.3% in the dark. A bioinspired machine-learning architecture can combine visual data with data from stretchable strain sensors to achieve human gesture recognition with high accuracy in complex environments.

Sub-full Model-Based Heterogeneous Sensor Fusion for Lateral State Estimation of Preceding Target Vehicles

Trajectory planning of an automated vehicle requires behavior knowledge of preceding target vehicles (PTVs), and lateral states, such as lateral velocity and yaw rate are key enablers for precise behavior description. However, lateral states of a PTV can hardly be measured directly by common onboard sensors. In addition, although these states can be transmitted via vehicle to vehicle (V2V), the accuracy is limited by the state holding process during the communication interval. Aiming at improved PTV lateral states, this article considers the estimation of these states using visual and V2V measurements, and proposes a three-stage fusion architecture consisting of input estimator, heterogeneous model-based local estimators, and fusion center. Specifically, to cope with the low rate of the received steering angle, the input estimator is constructed to obtain high rate control inputs. In the local estimator design, unlike the conventional modeling problems mixing control input errors with model errors in process noise, this paper separates them and develops a model adaption algorithm to compensate the time-varying covariance of the errors of the estimated control inputs. Then, to fuse the heterogeneous local estimates in the fusion center, a subfull model-based information matrix filter is designed to address this specific heterogeneous fusion problem, in which the local models are treated as submodels of a common full model. Hardware-in-the-loop experimental results show that the proposed method gives more accurate estimates in comparison with other approaches.

Human Activity Recognition from Multiple Sensors Data Using Multi-fusion Representations and CNNs

With the emerging interest in the ubiquitous sensing field, it has become possible to build assistive technologies for persons during their daily life activities to provide personalized feedback and services. For instance, it is possible to detect an individual’s behavioral pattern (e.g., physical activity, location, and mood) by using sensors embedded in smart-watches and smartphones. The multi-sensor environments also come with some challenges, such as how to fuse and combine different sources of data. In this article, we explore several methods of fusion for multi-representations of data from sensors. Furthermore, multiple representations of sensor data were generated and then fused using data-level, feature-level , and decision-level fusions . The presented methods were evaluated using three publicly available human activity recognition (HAR) datasets. The presented approaches utilize Deep Convolutional Neural Networks (CNNs). A generic architecture for fusion of different sensors is proposed. The proposed method shows promising performance, with the best results reaching an overall accuracy of 98.4% for the Context-Awareness via Wrist-Worn Motion Sensors (HANDY) dataset and 98.7% for the Wireless Sensor Data Mining (WISDM version 1.1) dataset. Both results outperform previous approaches.

LaserFlow: Efficient and Probabilistic Object Detection and Motion Forecasting

In this work, we present LaserFlow, an efficient method for 3D object detection and motion forecasting from LiDAR. Unlike the previous work, our approach utilizes the native range view representation of the LiDAR, which enables our method to operate at the full range of the sensor in real-time without voxelization or compression of the data. We propose a new multi-sweep fusion architecture, which extracts and merges temporal features directly from the range images. Furthermore, we propose a novel technique for learning a probability distribution over future trajectories inspired by curriculum learning. We evaluate LaserFlow on two autonomous driving datasets and demonstrate competitive results when compared to the existing state-of-the-art methods.

MNIST-NET10: A heterogeneous deep networks fusion based on the degree of certainty to reach 0.1% error rate. Ensembles overview and proposal

Ensemble methods have been widely used for improving the results of the best single classification model. A large body of works have achieved better performance mainly by applying one specific ensemble method. However, very few works have explored complex fusion schemes using heterogeneous ensembles with new aggregation strategies. This paper is three-fold: 1) It provides an overview of the most popular ensemble methods, 2) analyzes several fusion schemes using MNIST as guiding thread and 3) introduces MNIST-NET10, a complex heterogeneous fusion architecture based on a degree of certainty aggregation approach; it combines two heterogeneous schemes from the perspective of data, model and fusion strategy. MNIST-NET10 reaches a new record in MNIST with only 10 misclassified images. Our analysis shows that such complex heterogeneous fusion architectures based on the degree of certainty can be considered as a way of taking benefit from diversity.