Multiple Source Images Research Articles

Image splicing detection using low-dimensional feature vector of texture features and Haralick features based on Gray Level Co-occurrence Matrix

Digital image forgery has become hugely widespread, as numerous easy-to-use, low-cost image manipulation tools have become widely available to the common masses. Such forged images can be used with various malicious intentions, such as to harm the social reputation of renowned personalities, to perform identity fraud resulting in financial disasters, and many more illegitimate activities. Image splicing is a form of image forgery where an adversary intelligently combines portions from multiple source images to generate a natural-looking artificial image. Detection of image splicing attacks poses an open challenge in the forensic domain, and in recent literature, several significant research findings on image splicing detection have been described. However, the number of features documented in such works is significantly huge. Our aim in this work is to address the issue of feature set optimization while modeling image splicing detection as a classification problem and preserving the forgery detection efficiency reported in the state-of-the-art. This paper proposes an image-splicing detection scheme based on textural features and Haralick features computed from the input image’s Gray Level Co-occurrence Matrix (GLCM) and also localizes the spliced regions in a detected spliced image. We have explored the well-known Columbia Image Splicing Detection Evaluation Dataset and the DSO-1 dataset, which is more challenging because of its constituent post-processed color images. Experimental results prove that our proposed model obtains 95% accuracy in image splicing detection with an AUC score of 0.99, with an optimized feature set of dimensionality of 15 only.

Application of artificial intelligence digital holography technology based on medical sensors in the development of medical image fusion

With the rapid development of artificial intelligence and medical technology, sensor technology, as an important tool to obtain medical data, provides a wealth of physiological parameter information, but the traditional medical image fusion methods are difficult to make full use of the information provided by sensors. Therefore, this paper aims to explore the application of artificial intelligence digital holographic technology based on medical sensor in medical image fusion, and improve the accuracy and effect of medical image diagnosis and treatment. Based on the physiological parameter data obtained by medical sensor, combined with deep learning and image processing algorithm, a medical image fusion method based on artificial intelligence digital holography technology was proposed. The method uses the physiological parameter information provided by the sensor and combines multiple medical image sources to realize holographic medical image presentation. The experimental results show that the artificial intelligence digital holographic technology based on medical sensor can effectively improve the resolution and information content of medical image fusion. Compared with traditional medical image fusion methods, this method has higher image quality and more accurate diagnosis results.

Focus-aware and deep restoration network with transformer for multi-focus image fusion

Multi-focus image fusion (MFIF) aims to break the limitations of physical devices and integrate different focused regions from multiple source images into a single fully focused image. However, existing fusion methods suffer from two issues when designing the fusion framework: over-reliance on post-processing operations or underestimation of their impact, leading to unsatisfactory visual results. To this end, we propose a fusion model that combines focus-aware and deep restoration. Specifically, we first employ a designed focus-aware module to integrate spatial frequency information from various levels of depth features in the source images. This integration produces an initial decision map that reflects the focus information. Then, we treat the optimization of the decision map as an image restoration task, rather than relying on commonly used post-processing operations in the MFIF field. In the restoration task, the initial decision map is passed as input along with the gradient information of the source images to the deep restoration network combined with the transformer for obtaining the final decision map. To facilitate the training of the reconstruction network, we also propose a large dataset with labels and a new set of loss functions. Extensive experimental results conclusively demonstrate that the proposed method produces fusion results of superior visual quality. This is substantiated by an assessment across eight objective metrics, which encompass normalized mutual information (1.193), nonlinear correlation information entropy (0.846), gradient-based similarity measurement (0.727), phase congruency-based measurements (0.844), structural similarity-based assessments (0.902), human perception-inspired evaluations (0.808), peak signal-to-noise ratio (74.55), and correlation coefficients (0.971). These metrics collectively affirm the outstanding fusion performance of our method. The code will be able to be found at https://github.com/govenda/FARfusion.

A multi-channel neural network model for multi-focus image fusion

The objective of multi-focus image fusion (MFIF) is to generate a fully focused image through integrating multiple partially focused source images. Most of the existing methods do not fully consider the local gradient variation rate of the source image, which makes it difficult to accurately distinguish the small defocused (focused) region covered by the large focused (defocused) region. In addition, these methods also cause edge blurring because they do not take into account misregistration of the source images. To address these issues, in this paper, we propose a simple and effective multi-focus image fusion framework based on multi-channel Rybak neural network (MCRYNN) model. Specifically, the proposed MCRYNN model is highly sensitive to local gradient changes of images based on input receptive fields, which can process multiple source images in parallel and extract the features of focused regions. Moreover, the decision maps can accurately be generate in proposed method based on the information interaction effect of parallel network structure for multi-focus image fusion task. Finally, we conduct qualitative and quantitative experiments on public datasets, and the results show that the performance of the proposed method outperforms the state-of-the-art methods.

Probing Synergistic High-Order Interaction for Multi-modal Image Fusion.

Multi-modal image fusion aims to generate a fused image by integrating and distinguishing the cross-modality complementary information from multiple source images. While the cross-attention mechanism with global spatial interactions appears promising, it only captures second-order spatial interactions, neglecting higher-order interactions in both spatial and channel dimensions. This limitation hampers the exploitation of synergies between multi-modalities. To bridge this gap, we introduce a Synergistic High-order Interaction Paradigm (SHIP), designed to systematically investigate spatial fine-grained and global statistics collaborations between the multi-modal images across two fundamental dimensions: 1) Spatial dimension: we construct spatial fine-grained interactions through element-wise multiplication, mathematically equivalent to global interactions, and then foster high-order formats by iteratively aggregating and evolving complementary information, enhancing both efficiency and flexibility. 2) Channel dimension: expanding on channel interactions with first-order statistics (mean), we devise high-order channel interactions to facilitate the discernment of inter-dependencies between source images based on global statistics. We further introduce an enhanced version of the SHIP model, called SHIP++ that enhances the cross-modality information interaction representation by the cross-order attention evolving mechanism, cross-order information integration, and residual information memorizing mechanism. Harnessing high-order interactions significantly enhances our model's ability to exploit multi-modal synergies, leading in superior performance over state-of-the-art alternatives, as shown through comprehensive experiments across various benchmarks in two significant multi-modal image fusion tasks: pan-sharpening, and infrared and visible image fusion. The source code is publicly available at https://github.com/manman1995/HOIF.

UniDetector: Towards Universal Object Detection with Heterogeneous Supervision.

In this paper, we formally address universal object detection, which aims to detect every category in every scene. The dependence on human annotations, the limited visual information, and the novel categories in open world severely restrict the universality of detectors. We propose UniDetector, a universal object detector that recognizes enormous categories in the open world. The critical points for UniDetector are: 1) it leverages images of multiple sources and heterogeneous label spaces in training through image-text alignment, which guarantees sufficient information for universal representations. 2) it involves heterogeneous supervision training, which alleviates the dependence on the limited fully-labeled images. 3) it generalizes to open world easily while keeping the balance between seen and unseen classes. 4) it further promotes generalizing to novel categories through our proposed decoupling training manner and probability calibration. These contributions allow UniDetector to detect over 7k categories, the largest measurable size so far, with only about 500 classes participating in training. Our UniDetector behaves the strong zero-shot ability on large-vocabulary datasets - it surpasses supervised baselines by more than 5% without seeing any corresponding images. On 13 detection datasets with various scenes, UniDetector also achieves state-of-the-art performance with only a 3% amount of training data.

MFTSC: A Semantically Constrained Method for Urban Building Height Estimation Using Multiple Source Images

The use of remote sensing imagery has significantly enhanced the efficiency of building extraction; however, the precise estimation of building height remains a formidable challenge. In light of ongoing advancements in computer vision, numerous techniques leveraging convolutional neural networks and Transformers have been applied to remote sensing imagery, yielding promising outcomes. Nevertheless, most existing approaches directly estimate height without considering the intrinsic relationship between semantic building segmentation and building height estimation. In this study, we present a unified architectural framework that integrates the tasks of building semantic segmentation and building height estimation. We introduce a Transformer model that systematically merges multi-level features with semantic constraints and leverages shallow spatial detail feature cues in the encoder. Our approach excels in both height estimation and semantic segmentation tasks. Specifically, the coefficient of determination (R2) in the height estimation task attains a remarkable 0.9671, with a root mean square error (RMSE) of 1.1733 m. The mean intersection over union (mIoU) for building semantic segmentation reaches 0.7855. These findings underscore the efficacy of multi-task learning by integrating semantic segmentation with height estimation, thereby enhancing the precision of height estimation.

Robust multi-focus image fusion using focus property detection and deep image matting

Multi-focus image fusion techniques are designed to integrate the focused regions of multiple source images to produce an all-in-focus image. Unfortunately, the fusion results generated by existing methods are prone to suffer from chromatic aberrations and blurred boundary regions. In this paper, a multi-focus image fusion framework combining focus property detection and deep image matting is proposed to achieve high-quality fusion results. Specifically, we introduce spatial frequency detection blocks, which integrate the spatial frequency information of source images at different levels to achieve effective focus property detection for the initial decision. Then, the trimap acquired from the initial decision is optimized using a deep matting model so as to further detect the focused/defocused boundaries and obtain a more precise decision map. Meanwhile, a new hybrid loss function is introduced, and a large-scale dataset is created to improve the performance of the fusion framework. Finally, the fusion result is obtained by integrating the focused regions via the decision map. Compared to 18 other advanced fusion methods, the proposed method outperforms all of them in terms of the seven objective metrics evaluated on two publicly available datasets. These experimental results provide strong evidence that the proposed fusion framework excels not only in subjective vision but also in quantitative analysis. The code of the proposed method will be available at https://github.com/govenda/DIMF.

Towards robust registration of heterogeneous multispectral UAV imagery: A two-stage approach for cotton leaf lesion grading

Multiple source images acquired from diverse sensors mounted on unmanned aerial vehicles (UAVs) offer valuable complementary information for ground vegetation analysis. However, accurately aligning heterogeneous UAV images poses challenges due to differences in geometry, intensity, and noise resulting from varying imaging principles. This paper presents a two-stage registration method aimed at fusing visible RGB and multispectral images for cotton leaf lesion grading. The coarse alignment stage utilizes Scale Invariant Feature Transform (SIFT), while the refined alignment stage employs a novel correlation coefficient-based template matching. The proposed method first employs the EfficientDet network to detect infected cotton leaves with lesions in RGB images. Subsequently, lesion leaves in multiple spectral imagery (red, green, red edge, and near-infrared bands) are located using the perspective transformation matrix derived from SIFT and the coordinates of lesion leaves in RGB images. Refined registration between RGB and multispectral imagery is achieved through template matching with the new correlation coefficient. The registered reflectance data from the different spectral bands and RGB components are utilized to classify pixels in each infected leaf into lesion, healthy, and soil parts. The lesion grade is determined based on the ratio of lesion pixels to the total corresponding leaf area. Experimental results, compared with manual assessment, demonstrate a lesion leaves detection model with a mAP@0.5 of 91.01% and a leaf lesion grading accuracy of 92.01%. These results validate the suitability of the proposed method for UAV RGB and multispectral image registration, enabling automated cotton leaf lesion grading.

Multi-image blending holographic encryption system based on multi-source coherent diffraction and frequency-domain attention learning

The multi-image encryption method has the advantage of large information encryption capacity, which is conducive to the development of high-throughput cryptosystems. However, for multi-image encryption, the low PSNR caused by the crosstalk between images restricts the development of multi-image encryption methods. In our method, a novel optical multi-image encryption method based on multi-image coherent diffraction and frequency-domain attention learning is proposed. For encrypting, random aberrations brought about by diffraction are used instead of random phase masks to encrypt multiple source images into a single holographic ciphertext image based on holography and coherent diffraction imaging. A specially designed deep learning decryption network based on physical prior is proposed, which is used to resolve the source images of each channel respectively. Highlights and advantages include lower computational cost algorithm, higher real-time performance and better decryption quality. The feasibility, noise immunity, security and robustness of the proposed method are investigated through simulations and experiments. It is believed that the method we propose here can promote the further development of multi-optical image encryption systems.

Free-HeadGAN: Neural Talking Head Synthesis With Explicit Gaze Control

We present Free-HeadGAN, a person-generic neural talking head synthesis system. We show that modeling faces with sparse 3D facial landmarks is sufficient for achieving state-of-the-art generative performance, without relying on strong statistical priors of the face, such as 3D Morphable Models. Apart from 3D pose and facial expressions, our method is capable of fully transferring the eye gaze, from a driving actor to a source identity. Our complete pipeline consists of three components: a canonical 3D key-point estimator that regresses 3D pose and expression-related deformations, a gaze estimation network and a generator that is built upon the architecture of HeadGAN. We further experiment with an extension of our generator to accommodate few-shot learning using an attention mechanism, in case multiple source images are available. Compared to recent methods for reenactment and motion transfer, our system achieves higher photo-realism combined with superior identity preservation, while offering explicit gaze control.

TransFusion-net for multifocus microscopic biomedical image fusion

Background and objectiveDue to the depth of focus (DOF) limitations of the optical systems of microscopes, it is often difficult to achieve full clarity from microscopic biomedical images under high-magnification microscopy. Multifocus microscopic biomedical image fusion (MFBIF) can effectively solve this problem. Considering both information richness and visual authenticity, this paper proposes a transformer network for MFBIF called TransFusion-Net. MethodsTransFusion-Net consists of two modules. One module is an interlayer cross-attention module, which is used to obtain feature mappings under the long-range dependencies observed among multiple nonfocus source images. The other module is a spatial attention upsampling network (SAU-Net) module, which is used to obtain global semantic information after further spatial attention is applied. Thus, TransFusion-Net can simultaneously receive multiple input images from a nonfull-focus microscope and make full use of the strong correlations between the source images to output accurate fusion results in an end-to-end manner. ResultsThe fusion results were quantitatively and qualitatively compared with those of eight state-of-the-art algorithms. In the quantitative experiments, five evaluation metrics, QAB/F, QMI, QAVG, QCB, and PSNR, were used to evaluate the performance of each method, and the proposed method achieved values of 0.6574, 8.4572, 5.6305, 0.7341, and 89.5685, respectively, which are higher than those of the current state-of-the-art algorithms. In the qualitative experiments, a differential image was used for further validation, and the near-zero residuals visually verified the adequacy of the proposed method for fusion. Furthermore, we showed some fusion results of multifocused biomedical microscopy images to verify the reliability of the proposed method, which shows high-quality fusion results. ConclusionMultifocus biomedical microscopic image fusion can be accurately and effectively achieved by devising a deep convolutional neural network with joint cross-attention and spatial attention mechanisms.

Adaptive adversarial prototyping network for few-shot prototypical translation

Translating multiple real-world source images to a single prototypical image is a challenging problem. Notably, these source images belong to unseen categories that did not exist during model training. We address this problem by proposing an adaptive adversarial prototype network (AAPN) and enhancing existing one-shot classification techniques. To overcome the limitations that traditional works cannot extract samples from novel categories, our method tends to solve the image translation task of unseen categories through a meta-learner. We train the model in an adversarial learning manner and introduce a style encoder to guide the model with an initial target style. The encoded style latent code enhances the performance of the network with conditional target style images. The AAPN outperforms the state-of-the-art methods in one-shot classification of brand logo dataset and achieves the competitive accuracy in the traffic sign dataset. Additionally, our model improves the visual quality of the reconstructed prototypes in unseen categories. Based on the qualitative and quantitative analysis, the effectiveness of our model for few-shot classification and generation is demonstrated.

FEPVNet: A Network with Adaptive Strategies for Cross-Scale Mapping of Photovoltaic Panels from Multi-Source Images

The world is transitioning to renewable energy, with photovoltaic (PV) solar power being one of the most promising energy sources. Large-scale PV mapping provides the most up-to-date and accurate PV geospatial information, which is crucial for planning and constructing PV power plants, optimizing energy structure, and assessing the ecological impact of PVs. However, previous methods of PV extraction relied on simple models and single data sources, which could not accurately obtain PV geospatial information. Therefore, we propose the Filter-Embedded Network (FEPVNet), which embeds high-pass and low-pass filters and Polarized Self-Attention (PSA) into a High-Resolution Network (HRNet) to improve its noise resistance and adaptive feature extraction capabilities, ultimately enhancing the accuracy of PV extraction. We also introduce three data migration strategies by combining Sentinel-2, Google-14, and Google-16 images in varying proportions and transferring the FEPVNet trained on Sentinel-2 images to Gaofen-2 images, which improves the generalization performance of models trained on a single data source for extracting PVs in images of different scales. Our model improvement experiments demonstrate that the Intersection over Union (IoU) of FEPVNet in segmenting China PVs in Sentinel-2 images reaches 88.68%, a 2.37% increase compared to the HRNet. Furthermore, we use FEPVNet and the optimal migration strategy to extract photovoltaics across scales, achieving a precision of 94.37%. In summary, this study proposes the FEPVNet model with adaptive strategies for extracting PVs from multiple image sources, with significant potential for application in large-scale PV mapping.

Local augmentation based consistency learning for semi-supervised pathology image classification

Background and objectiveLabeling pathology images is often costly and time-consuming, which is quite detrimental for supervised pathology image classification that relies heavily on sufficient labeled data during training. Exploring semi-supervised methods based on image augmentation and consistency regularization may effectively alleviate this problem. Nevertheless, traditional image-based augmentation (e.g., flip) produces only a single enhancement to an image, whereas combining multiple image sources may mix unimportant image regions resulting in poor performance. In addition, the regularization losses used in these augmentation approaches typically enforce the consistency of image level predictions, and meanwhile simply require each prediction of augmented image to be consistent bilaterally, which may force pathology image features with better predictions to be wrongly aligned towards the features with worse predictions. MethodsTo tackle these problems, we propose a novel semi-supervised method called Semi-LAC for pathology image classification. Specifically, we first present local augmentation technique to randomly apply different augmentations produces to each local pathology patch, which can boost the diversity of pathology image and avoid mixing unimportant regions in other images. Moreover, we further propose the directional consistency loss to enforce restrictions on the consistency of both features and prediction results, thus improving the ability of the network to obtain robust representations and achieve accurate predictions. ResultsThe proposed method is evaluated on Bioimaging2015 and BACH datasets, and the extensive experiments show the superior performance of our Semi-LAC compared with state-of-the-art methods for pathology image classification. ConclusionsWe conclude that using the Semi-LAC method can effectively reduce the cost for annotating pathology images, and enhance the ability of classification networks to represent pathology images by using local augmentation techniques and directional consistency loss.

Radiographic Parameters of Adult Hip Dysplasia.

As knowledge about the origin and morphologic characteristics of hip pain in the young adult has evolved, so too has the clinician's ability to assess for various pathologies of the hip on radiographs, magnetic resonance imaging (MRI)/magnetic resonance arthrography (MRA), and computed tomography (CT). Because there is no algorithm at this time directly indicating what to do in more subtle hip morphologies, such as microinstability and borderline hip dysplasia (BHD), a skilled hip preservation specialist must use multiple imaging sources and know how to interpret them correctly. Imaging parameters used in the workup for hip dysplasia and BHD include the lateral center-edge angle, Tönnis angle, iliofemoral line, and presence of an upsloping lateral sourcil or everted labrum, among many others. The purpose of this narrative review was to detail various established criteria and parameters on anteroposterior pelvis plain radiographs, MRI/MRA, and CT that assist in defining the nature and severity of instability present in a dysplastic hip, thereby aiding in the development of patient-specific surgical treatment plans.

Towards faithful neural fusion to infrared and visible images with a full-scale connected network

The aim of image fusion is to merge image information from multiple source images into one image. The fused image can contain more complete and clear scene information without introducing artifacts. The results of image fusion are required to be faithful to the original image contents. However, we notice that the unfaithful fusions often contain hallucinated content, which cannot be aligned with any input images. For the first time, we propose a full-scale connected-based fusion network (FSCF-Net) for infrared and visible image fusion to achieve the goal. To make full use of the multiscale features of the image, the full-scale skip connection is used in the decoder network of FSCF-NET to fuse the full-scale deep features extracted by the encoder. To enforce faithfulness, the FSCF-Net adopts a hybrid loss function consisting of detail-preserving loss and structure-preserving loss. To effectively fuse the features of different source images, a spatial-channel attention-based feature fusion strategy (SCAF) is proposed. The SCAF fusion strategy measures the importance of features from both spatial and channel dimensions. The detailed analysis is provided about the proposed model in our experiments that are performed on publicly available datasets. We adopt subjective evaluation and 6 objective metrics to compare the proposed FSCF-Net with 10 state-of-the-art fusion algorithms. The experiments demonstrate that our method achieves the best comprehensive performance on the overall metrics.

Locating and Imaging Fiber Breaks in CFRP Using Guided Wave Tomography and Eddy Current Testing.

In this paper, guided Lamb wave tomography and eddy current testing (ECT) techniques were combined to locate and evaluate fiber breaks in carbon-fiber-reinforced plastic (CFRP) structures. Guided wave testing (GWT) and computed tomography (CT) imaging were employed to quickly locate fiber breaks in the CFRP plate. From B-scans performed along two different fiber orientations (0 and 90 degrees), parallel-beam projections of different features were extracted from the guided wave signals, using signal-processing techniques (such as wavelet and Hilbert transforms) and statistical functions (such as skewness and kurtosis). The parallel-beam projections of each individual feature were used as input in computed tomography imaging reconstruction to approximately estimate the location of fiber breaks. From the obtained reconstructed images, image-fusion techniques were applied to get complementary information from multiple source images into one single image. After locating the fiber breaks, C-scans were performed in the vicinity of the damage, using an ECT probe with double excitation configuration to evaluate the condition of the fiber break.

Improved Procedure for Multi-Focus Images Using Image Fusion with qshiftN DTCWT and MPCA in Laplacian Pyramid Domain

Multi-focus image fusion (MIF) uses fusion rules to combine two or more images of the same scene with various focus values into a fully focused image. An all-in-focus image refers to a fully focused image that is more informative and useful for visual perception. A fused image with high quality is essential for maintaining shift-invariant and directional selectivity characteristics of the image. Traditional wavelet-based fusion methods, in turn, create ringing distortions in the fused image due to a lack of directional selectivity and shift-invariance. In this paper, a classical MIF system based on quarter shift dual-tree complex wavelet transform (qshiftN DTCWT) and modified principal component analysis (MPCA) in the laplacian pyramid (LP) domain is proposed to extract the focused image from multiple source images. In the proposed fusion approach, the LP first decomposes the multi-focus source images into low-frequency (LF) components and high-frequency (HF) components. Then, qshiftN DTCWT is used to fuse low and high-frequency components to produce a fused image. Finally, to improve the effectiveness of the qshiftN DTCWT and LP-based method, the MPCA algorithm is utilized to generate an all-in-focus image. Due to its directionality, and its shift-invariance, this transform can provide high-quality information in a fused image. Experimental results demonstrate that the proposed method outperforms many state-of-the-art techniques in terms of visual and quantitative evaluations.

Off-Road Navigation Maps for Robotic Platforms using Convolutional Neural Networks

Abstract. As part of AMADEE-20, an integrated Mars analog field mission in the Negev Desert in Israel conducted by the Austrian Space Forum, an exploration cascade for the remote sensing of extraterrestrial terrain was implemented. For this purpose, aerial robots were conceptualized, which were used in an iterative process to generate a navigational map for an autonomous ground vehicle. This work presents the process for generating navigation maps using multiple aerial image sources from satellites as well as from low orbiting aerial vehicles. First, Deep Learning methods are used to analyze a high altitude aerial images of a large area, creating a basis map for mission planning and navigation. Second, high resolution unmanned aerial vehicle (UAV) images were recorded on low altitude for a pre-defined area of interest, processed with Deep-Learning and Structure from Motion and used to update the basis map. This approach results in a high accuracy navigation map for autonomous, off-road robot navigation. Experiments during the AMADEE-20 mission in the Israeli Negev Desert validated the proposed methods by sending an autonomous ground vehicle through the environment using the generated map.