Fusion-based Approach Research Articles

A Multi-modal Feature Fusion-based Approach for Mobile Application Classification and Recommendation

<p>With the rapid growth of the number and type of mobile applications, it becomes challenging to accurately classify and recommend mobile applications according to users’ individual requirements. The existing mobile application classification and recommendation methods, for one thing, do not take into account the correlation between large-scale data and model. For another, they also do not fully exploit the multi-modal, fine-grained interaction features with high-order and low-order in mobile application. To tackle this problem, we propose a mobile application classification and recommendation method based on multi-modal feature fusion. The method firstly extracts the image and description features of the mobile application using an “involution residual network + pre-trained language representation” model (i.e. the TRedBert model). Afterwards, these features are fused by using the attention mechanism in the transformer model. Then, the method classifies the mobile applications based on the fused features through a softmax classifier. Finally, the method extracts the high-order and low-order embedding features of the mobile app with a bi-linear feature interaction model (FiBiNET) based on the classification results of the mobile app, by combining the Hadamard product and inner product to achieve fine-grained high-order and low-order feature interaction, to update the mobile app representation and complete the recommendation task. The multiple sets of comparison experiments are performed on Kaggle’s real dataset, i.e., 365K IOS Apps Dataset. And the experimental results demonstrated that the proposed approach outperforms other methods in terms of Macro F1, Accuracy, AUC and Logloss.</p> <p> </p>

Fusion and integration of heterogeneous close range remote sensing and geophysical data. The case of Grumentum.

Abstract This paper deals with the integration and fusion of UAV-based imagery, including Vis-NIR multispectral and infrared thermography data, with georadar and geomagnetic prospecting, applied on Grumentum, pre-Roman and Roman city, with an important phase dated to Late Antiquity, of the ancient Lucania, in Basilicata region. The data were processed individually and then integrated with each other in order to recover as much information as possible from the different signals. Due to the qualitative and quantitative approaches to the integration and fusion of the diverse data sets and features, it was possible to detect some urban blocks (insulae) reasonably referable to the Roman age, although obviously not datable, and other features spatially linked to the late antique/early medieval church. Grumentum’s experience shows considerable potential as well as limits in the use of the image fusion-based approach for the enhancement of archaeological features, from which emerges the need to continue experimenting these approaches on a greater number of case studies.

Lithium-Ion Power Battery Grouping: A Multisource Data Fusion-Based Clustering Approach and Distributed Deployment

Abstract Consistence of lithium-ion power battery significantly affects the life and safety of battery modules and packs. To improve the consistence, battery grouping is employed, assembling batteries with similar electrochemical characteristics to make up modules and packs. Therefore, grouping process boils down to unsupervised clustering problem. Current used grouping approaches include two aspects, static characteristics based and dynamic based. However, there are three problems. First, the common problem is under utilization of multi-source data. Second, for the static characteristics based, there is grouping failure over time. Third, for the dynamic characteristics based, there is high computational complexity. To solve these problems, we propose a distributed multisource data fusion based battery grouping approach. The proposed approach designs an effective network structure for multisource data fusing and feature extracting from both static and dynamic multisource data. We apply our approach on real battery modules and record state of health (SOH) during charging-discharging cycles. Experiments indicate that the proposed approach can increase SOH of modules by 3.89% and reduce the inconsistence by 68.4%. Meanwhile, with the distributed deployment the time cost is reduced by 87.9% than the centralized scheme.

Applying a non-invasive multi-spectral sensing technique to two-phase flow measurements for pipeline monitoring

"Smart" sensors, with a fusion-based approach, pave the way for potentially changing how pipeline network systems are controlled, maintained, and monitored. This is typically done through structural modeling or spot monitoring using micro-electromechanical systems, strain gauges or through general conventional sensing. One of the challenges facing new developments in structural health monitoring (SHM) of a deployed asset is application to inaccessible locations. This work presents the use of a non-intrusive integrity monitoring sensor system utilising a multi-spectral approach to give more trusted insights to structural integrity as well as flow characteristics. The paper addresses the fundamental problems in identifying and characterising two-phase (air–water and oil–water) flow patterns and flow pattern transitions using numerous sensing techniques: strain gauges, optical Fibre Bragg Gratings, thermocouples (K-type), accelerometers, acoustic emission analysis and gyroscopes. By providing low noise, un-biased strain measurements, the non-intrusive sensor system can verify and improve integrity monitoring, providing a reliable alternative to conventional sensing. In addition, the non-intrusive sensor system enables pressure monitoring, providing flow assurance data and, provides good data integrity, making it an ideal tool as an enhanced surveillance strategy in real time for SHM of pipeline networks.

Sensor fusion-based approach for the field robot localization on Rovitis 4.0 vineyard robot

Sensor fusion-based approach for the field robot localization on Rovitis 4.0 vineyard robot

A multi-representation re-ranking model for Personalized Product Search

In recent years, a multitude of e-commerce websites arose. Product Search is a fundamental part of these websites, which is often managed as a traditional retrieval task. However, Product Search has the ultimate goal of satisfying specific and personal user needs, leading users to find and purchase what they are looking for, based on their preferences. To maximize users’ satisfaction, Product Search should be treated as a personalized task. In this paper, we propose and evaluate a simple yet effective personalized results re-ranking approach based on the fusion of the relevance score computed by a well-known ranking model, namely BM25, with the scores deriving from multiple user/item representations. Our main contributions are: (1) we propose a score fusion-based approach for personalized re-ranking that leverages multiple user/item representations, (2) our approach accounts for both content-based features and collaborative information (i.e. features extracted from the user–item interactions graph), (3) the proposed approach is fast and scalable, can be easily added on top of any search engine and it can be extended to include additional features. The performed comparative evaluations show that our model can significantly increase the retrieval effectiveness of the underlying retrieval model and, in the great majority of cases, outperforms modern Neural Network-based personalized retrieval models for Product Search.

Region of interest enhancement for backlit images using global tone mappings and fusion

ABSTRACT Backlit images simultaneously include dark as well as light regions of intensity. Objects in dark and low-contrast regions are complex for humans to perceive. In particular, such parts are selected as the image's region of interest (ROI). In backlit images, ROI is typically very dark and of low contrast, which makes visualization difficult. This paper explores and presents an approach for contrast enhancement and preserving the naturalness of the ROI of the backlit image. The proposed methodology considers stretching the contrast and balancing other aspects of the ROI, such as textural details, visibility, etc. In the last stage, a fusion-based approach is carried out to obtain enhanced ROI of the backlit image. It also examines the experimental results of the proposed method using various steps and, based on the validation, confirms that the proposed solution produces better results for the ROI of the backlit image than the other state-of-the-art methods.

Fusion-based damage diagnostics for stiffened composite panels

Conducting damage diagnostics on stiffened panels is commonly performed using a single SHM technique. However, each SHM technique has both its strengths and limitations. Rather than straining the expansion of single SHM techniques going beyond their intrinsic capacities, these strengths and limitations should instead be considered in their application. In this work, we propose a novel fusion-based methodology between data from two SHM techniques in order to surpass the capabilities of a single SHM technique. The aim is to show that by considering data fusion, a synergy can be obtained, resulting in a comprehensive damage assessment, not possible using a single SHM technique. For this purpose, three single-stiffener carbon–epoxy panels were subjected to fatigue compression after impact tests. Two SHM techniques monitored damage growth under the applied fatigue loads: acoustic emission and distributed fiber optic strain sensing. Four acoustic emission sensors were placed on each panel, thereby allowing for damage detection, localization, type identification (delamination), and severity assessment. The optical fibers were adhered to the stiffener feet’ surface, and its strain measurements were used for damage detection, disbond localization, damage type identification (stiffness degradation and disbond growth), and severity assessment. Different fusion techniques are presented in order to integrate the acoustic emission and strain data. For damage detection and severity assessment, a hybrid health indicator is obtained by feature-level fusion while a complementary and cooperative fusion of the diagnostic results is developed for damage localization and type identification. We show that damage growth can be monitored up until final failure, thereby performing a simultaneous damage assessment on all four SHM levels. In this manner, we demonstrate that by proposing a fusion-based approach toward SHM of composite structures, the intrinsic capacity of each SHM technique can be utilized, leading to synergistic effects for damage diagnostics.

Cutting tool temperature monitoring in circular sawing: measurement and multi-sensor feature fusion-based prediction

In this study, contact thermocouple temperature measurements were used in the wood circular sawing process, and the saw temperature was monitored at two locations—one close to the cutting edge and the other close to the blade center. The effects of depth of cut, rotation speed, and feed speed on the tool temperature were analyzed and discussed. Also, due to the complexity of tool temperature measurement during the machining process, the study proposed a sensor fusion-based approach for indirect monitoring of the tool temperature. Accordingly, the cutting process was monitored with power, sound, vibration, and acoustic emission (AE) sensors, and the extracted sensory features were used to predict the saw temperature by training random forest models. The results showed that the cutting parameters significantly impact the tool temperature, and the rotation speed has the most complex and nonlinear effect on the blade temperature. The role of rotation speed on the tool temperature is governed by both the kinematics of the machining process and chip size along with the dynamic behavior of the saw blade and the proximity of the rotation speed to the tool critical and flutter speeds. Among the four investigated sensors, AE showed better performance for tool temperature monitoring and, combined with the vibration signals, yielded the best monitoring performance. The developed sensory models can handle the challenges associated with tool temperature measurement/prediction and, at the same time, can be used for monitoring other aspects of machining processes, including surface quality.

Application of Multisensor Information Fusion Technology in the Measurement of Dynamic Machining Errors of Computer Numerical Control (CNC) Machine Tools

A CNC machine tool is process control equipment integrating machine, electricity, and liquid, which makes its fault diagnosis complex and special due to its own advanced, complex, and intelligent characteristics. Traditional diagnostic methods rely on the engineering experience of technical personnel, which incorporates human subjective factors, and can only perform qualitative analysis, resulting in low diagnostic efficiency. And through a single sensor to detect and diagnose the machine tool, the accuracy and credibility of the decision are low, and the system is also weak against interference. In this paper, we first summarize the composition and working principle of CNC machine tools and analyze the working condition signals generated by CNC machine tools and the sensors that collect the signals and decide to use a multisensor multisignal fusion‐based approach to monitor the machine tool status. It is possible to obtain more effective and valuable information from the observed information through multiple sensors so that the goal of fusion can be achieved. In this paper, a multisensor fusion technique based on wavelet transform and neural network fusion is applied to a machine tool condition monitoring system. The theoretical basis of wavelet analysis and neural network is introduced, and the composition of the condition monitoring system and the process of applying multisensor fusion technology based on wavelet analysis and neural network in the condition monitoring system are given. A complete software and hardware system for online monitoring of CNC machine tools is established. In order to improve the accuracy of the mathematical model, the use of a neural network to fit the nonlinear data and the use of coarse set theory to simplify the relevant data can effectively solve the accurate establishment of the mathematical model in the error compensation method. The thermal error compensation method for CNC machine tools is proposed based on rough set theory, ant colony algorithm, and neural network. This paper first investigates the current development of error compensation technology for CNC machining centers, analyzes the various error sources of CNC machine tools, and finds out the influencing factors affecting the errors of CNC machine tools.

Fusion of Handcrafted and Deep Features for Forgery Detection in Digital Images

Content authentication of digital images has captured the attention of forensic experts and security researchers due to a multi-fold increase in the dissemination of multimedia data through the open and vulnerable Internet. Shrewd attackers successfully devise novel ways to challenge state-of-art forensic tools used for forgery detection in digital images. Feature engineering approaches have yielded up to 97% accuracy on benchmarked datasets. Deep learning approaches have shown promising results in various image classification problems but cannot find hidden patterns in digital images, which can reliably detect image forgeries. State-of-art accuracy of deep learning approaches for forgery detection is up to 98% on benchmarked datasets. The objective of the proposed approach is to further escalate the detection accuracy, pushing it near 100%. In this paper, a synergy of handcrafted features based on color characteristics and deep features using the image’s luminance channel is employed to mine patterns responsible for accurate forgery detection. In the first Stream, 648-D Markov-based features are computed from the quaternion discrete cosine transform of the image. In the second Stream, the luminance channel of YCbCr colorspace is used to extract the Local Binary Pattern of the image. Further, local binary feature maps are fed to the pre-trained ResNet-18 model to obtain a 512-D feature vector called ’ResFeats’ from the last layer of the model’s convolutional base portion. The handcrafted features from Stream I and ResFeats from Stream II are combined to form an 1160-D feature vector. Further, classification is performed using a shallow neural network, and the method is tested on CASIA v1 and CASIA v2 datasets. The accuracy of the proposed fusion-based approach is 99.3% on benchmark datasets.

Sensor Fusion-Based Approach to Eliminating Moving Objects for SLAM in Dynamic Environments.

Accurate localization and reliable mapping is essential for autonomous navigation of robots. As one of the core technologies for autonomous navigation, Simultaneous Localization and Mapping (SLAM) has attracted widespread attention in recent decades. Based on vision or LiDAR sensors, great efforts have been devoted to achieving real-time SLAM that can support a robot’s state estimation. However, most of the mature SLAM methods generally work under the assumption that the environment is static, while in dynamic environments they will yield degenerate performance or even fail. In this paper, first we quantitatively evaluate the performance of the state-of-the-art LiDAR-based SLAMs taking into account different pattens of moving objects in the environment. Through semi-physical simulation, we observed that the shape, size, and distribution of moving objects all can impact the performance of SLAM significantly, and obtained instructive investigation results by quantitative comparison between LOAM and LeGO-LOAM. Secondly, based on the above investigation, a novel approach named EMO to eliminating the moving objects for SLAM fusing LiDAR and mmW-radar is proposed, towards improving the accuracy and robustness of state estimation. The method fully uses the advantages of different characteristics of two sensors to realize the fusion of sensor information with two different resolutions. The moving objects can be efficiently detected based on Doppler effect by radar, accurately segmented and localized by LiDAR, then filtered out from the point clouds through data association and accurate synchronized in time and space. Finally, the point clouds representing the static environment are used as the input of SLAM. The proposed approach is evaluated through experiments using both semi-physical simulation and real-world datasets. The results demonstrate the effectiveness of the method at improving SLAM performance in accuracy (decrease by 30% at least in absolute position error) and robustness in dynamic environments.

Robust 3D Hand Detection from a Single RGB-D Image in Unconstrained Environments

Three-dimensional hand detection from a single RGB-D image is an important technology which supports many useful applications. Practically, it is challenging to robustly detect human hands in unconstrained environments because the RGB-D channels can be affected by many uncontrollable factors, such as light changes. To tackle this problem, we propose a 3D hand detection approach which improves the robustness and accuracy by adaptively fusing the complementary features extracted from the RGB-D channels. Using the fused RGB-D feature, the 2D bounding boxes of hands are detected first, and then the 3D locations along the z-axis are estimated through a cascaded network. Furthermore, we represent a challenging RGB-D hand detection dataset collected in unconstrained environments. Different from previous works which primarily rely on either the RGB or D channel, we adaptively fuse the RGB-D channels for hand detection. Specifically, evaluation results show that the D-channel is crucial for hand detection in unconstrained environments. Our RGB-D fusion-based approach significantly improves the hand detection accuracy from 69.1 to 74.1 comparing to one of the most state-of-the-art RGB-based hand detectors. The existing RGB- or D-based methods are unstable in unseen lighting conditions: in dark conditions, the accuracy of the RGB-based method significantly drops to 48.9, and in back-light conditions, the accuracy of the D-based method dramatically drops to 28.3. Compared with these methods, our RGB-D fusion based approach is much more robust without accuracy degrading, and our detection results are 62.5 and 65.9, respectively, in these two extreme lighting conditions for accuracy.

Quality-driven tone-mapping operator: a pseudo-exposure fusion-based approach

High dynamic range (HDR) images can present realistic scenes with improved detail in high- and low-brightness regions. Due to their limitations in application, HDR images are often processed by tone-mapping operators (TMOs) to be backward-compatible with traditional devices. Hence, for a better visual experience, a TMO should naturally preserve the original scene information with image quality as a guideline. With such motivation, we propose a new quality-driven TMO (QdTMO) by using a pseudo-exposure scheme and a local optimal exposed image (LOEI) fusion network. First, a pseudo-exposure scheme is designed to generate richly detailed LOEIs for different brightness regions of the image. Then, scene information in the original HDR image can be naturally preserved by a specifically designed LOEIs fusion network. Finally, the exposure residual energy and brightness naturalness are used to optimize the pseudo-exposure scheme and LOEI fusion for better quality in the resulting tone-mapped image. Experimental results demonstrate that tone-mapped images generated by the proposed QdTMO have better performance than existing TMOs in both subjective and objective evaluations.

Fusion-Based Hypoxia Estimates: Combining Geostatistical and Mechanistic Models of Dissolved Oxygen Variability.

The need to characterize and track coastal hypoxia has led to the development of geostatistical models based on in situ observations of dissolved oxygen (DO) and mechanistic models based on a representation of biophysical processes. To integrate the benefits of these two distinct modeling approaches, we develop a space-time geostatistical framework for synthesizing DO observations with hydrodynamic-biogeochemical model simulations and meteorological time series (as covariates). This fusion-based approach is used to estimate hypoxia in the northern Gulf of Mexico across summers from 1985 to 2017. Deterministic trends with dynamic covariates explain over 35% of the variability in DO. Moreover, cross-validation results indicate that 58% of DO variability is explained when combining these trends with spatiotemporal interpolation, which is substantially better than mechanistic or conventional geostatistical hypoxia modeling alone. The fusion-based approach also reduces hypoxic area uncertainties by 11% on average and up to 40% in months with sparse sampling. Moreover, our new estimates of mean summer hypoxic area changed by >10% in a majority of years, relative to previous geostatistical estimates. These fusion-based estimates can be a valuable resource when assessing the influence of hypoxia on the coastal ecosystem.

Catheter navigation support for liver radioembolization guidance: feasibility of structure-driven intensity-based registration.

The fusion of pre/intraoperative images may improve catheter manipulation during radioembolization (RE) interventions by adding relevant information. The objective of this work is to propose and evaluate the performance of a RE guidance strategy relying on structure-driven intensity-based registration between preoperative CTA and intraoperative X-ray images. The navigation strategy is decomposed into three image fusion steps, supporting the catheter navigation from the femoral artery till reaching the injection site (IS). During the pretreatment assessment intervention, the aorta and the origins of its side branches are projected on the intraoperative 2D fluoroscopy following a 3D/2D bone-based registration process, to assist the celiac trunk access. Subsequently, a similar approach consisting in projecting the hepatic vasculature on intraoperative DSA through 3D/2D vessel-based registration is performed to assist the IS location. Lastly, the selected IS is reproduced during the treatment intervention by employing 2D/2D image-based registration between pretreatment and treatment fluoroscopic images. The three fusion steps were independently evaluated on subsets of 20, 19 and 5 patient cases, respectively. Best results were obtained with gradient difference as similarity measure and with a delimited preoperative vascular structure for vessel-based registration. The approach resulted in qualitatively appropriate anatomical correspondences when projecting the preoperative structures on intraoperative images. With the best configuration, the registration steps showed accuracy and feasibility in aligning data, with global mean landmarks errors of 1.59mm, 2.32mm and 2.17mm, respectively, a computation time that never exceeded 5s, 25s and 11s, respectively, and a user interaction limited to manual initialization of the 3D/2D registration. An image fusion-based approach has been specifically proposed for RE procedures guidance. The catheter manipulation strategy based on the fusion of pre- and intraoperative images has the potential to support different steps of the RE clinical workflow and to guide the overall procedure.

Underwater Image Restoration Based on Image Fusion

The haziness in underwater images occurs due to two major phenomena namely absorption and scattering of light. Hence, we have proposed an image fusion-based approach to improve the visibility of images obtained underwater. The proposed method uses a single hazy image. Initially the colour corrected and contrast improved versions of the image are obtained. Further, Laplace transform is applied which is followed by replication and saliency mapping on each of the derived images. Multi-scale image fusion technique has been used to combine the inputs. This enables each of the fused images to contribute the most essential feature to obtain the resultant image. Thus, the proposed method significantly restores the quality of the input distorted images

Iris Liveness Detection Using Fusion of Domain-Specific Multiple BSIF and DenseNet Features.

In the past few years, some fusion-based approaches have been proposed to constitute discriminatory features for iris liveness detection. However, several methods exist in the literature for iris feature extraction and, thus, identifying an optimal composite of such features is still a vital challenge. This article also proposes a score-level fusion of two distinct domain-specific features, i.e., multiple binarized statistical image feature (BSIF) and DenseNet-based features. However, instead of randomly scrutinizing such features, statistical tests are executed on six predominant iris features to identify the optimal feature set to combine. Particularly, this work emphasizes textured-lens-based presentation attacks and aims to identify the type of contact lenses within the iris samples. The experimental analysis depicts that the domain-specific features substantially outperform the generic features while discriminating live iris from the artifacts. Furthermore, the proposed fusion-based approach is assessed on three iris datasets and the outcomes are compared with various state of the arts using three validation protocols in terms of equal error rate (EER). The comparative analysis perceived that the proposed method obtains a significant performance gain over the existing approaches and offers an improved benchmark for both, iris liveness detection and contact lens identification.

Double Stage Gaussian Filter for Better Underwater Image Enhancement

Major issue in marine environment imaging is the expulsion of hazy scenes caused by natural phenomena such as absorption and scattering in underwater images. The visual appearance of the underwater image needs to be improved. In this paper, a double stage Gaussian filter (DSGF) and fusion-based approach to enhance the underwater images is proposed. In the proposed two stage process for image fusion, first section deals with applying the DSGF with high, low, and medium sigma values on the individual R, G and B components of input RGB images and second section deals with applying colour component wise contrast limited adaptive histogram followed by median filtering. The outputs of these two sections are fused using principal component analysis fusion technique. Fused image is post-processed using the colour contrast correction for better visibility results. Proposed method is compared with single stage Gaussian filter where only one stage of Gaussian filtering is applied for reduced computation, Adaptive histogram equalization and histogram specification methods. Proposed method results in quality images obtained through gradual intensity improvement and contrast correction process. Qualitative and quantitative analysis are performed on the enhanced underwater images to observe the effectiveness of the algorithm used. The performance measures such as entropy, SSIM, and AMBE unveil the verification of the proposed findings.

Metabolic enzyme clustering by coiled coils improves the biosynthesis of resveratrol and mevalonate

The clustering of biosynthetic enzymes is used in nature to channel reaction products and increase the yield of compounds produced by multiple reaction steps. The coupling of multiple enzymes has been shown to increase the biosynthetic product yield. Different clustering strategies have particular advantages as the spatial organization of multiple enzymes creates biocatalytic cascades with a higher efficiency of biochemical reaction. However, there are also some drawbacks, such as misfolding and the variable stability of interaction domains, which may differ between particular biosynthetic reactions and the host organism. Here, we compared different protein-based clustering strategies, including direct fusion, fusion mediated by intein, and noncovalent interactions mediated through small coiled-coil dimer-forming domains. The clustering of enzymes through orthogonally designed coiled-coil interaction domains increased the production of resveratrol in Escherichia coli more than the intein-mediated fusion of biosynthetic enzymes. The improvement of resveratrol production correlated with the stability of the coiled-coil dimers. The coiled-coil fusion-based approach also increased mevalonate production in Saccharomyces cerevisiae, thus demonstrating the wider applicability of this strategy.