Multiple Image Pairs Research Articles

Confocal microscopy multi-focus image fusion method based on axial information guidance.

Aiming at the problems of poor anti-interference of existing pixel-level fusion rules and low efficiency of transform domain fusion rules, this study proposes a confocal microscopic multi-focus image fusion method (IGCM) based on differential confocal axial information guidance. Unlike traditional multi-focus image fusion (MFIF) methods, IGCM uses height information rather than grayscale or frequency to determine clear areas. First, the differential confocal axial measurement curve is calibrated to determine the suitable scan step u. Second, the image set required for fusion is constructed by performing a hierarchical scan of the measurement samples. Then, multiple differential image pairs are constructed using the step size u and the set of images, and the extraction area of the current reference image is decided based on the height obtained from the differential image. Finally, the regions determined by each reference image are extracted and the duplicated pixels are averaged to obtain the MFIF image. The results were that IGCM improves the interference immunity based on pixel-level image fusion compared to the maximum peak fusion method. Compared with other MFIFs, IGCM has excellent fusion efficiency while ensuring fusion clarity, which can meet the application scenario of real-time fusion and offers a new approach to panoramic depth images for confocal devices.

USIS: A unified semantic image synthesis model trained on a single or multiple samples

Semantic image synthesis methods learn to generate new images conditioned on predefined semantic label maps. Existing methods require access to large-volume samples labeled with semantic maps, which limits their applications. We propose USIS, a Unified Semantic Image Synthesis model which can be trained on only a single or multiple pairs of images and semantic maps. Once trained, a USIS model can generate new images according to unseen semantic maps, as existing semantic image synthesis methods do. Specifically, we design a hierarchical architecture to reconstruct training samples and gradually learn the distributions of multi-scale patches in samples from coarse to fine. To avoid the error accumulation across scales, we propose a mixed training strategy to stabilize the training process. Extensive experiments on one- or multiple-sample datasets show our proposed model achieves state-of-the-art performance in terms of visual fidelity.

PM-Net: A Multi-Level Keypoints Detector and Patch Feature Learning Network for Optical and SAR Image Matching

Due to the differences in radiation and geometric characteristics of optical and synthetic aperture radar (SAR) images, there is still a huge challenge for accurate matching. In this paper, we propose a patch-matching network (PM-Net) to improve the matching performance of optical and SAR images. First, a multi-level keypoints detector (MKD) with fused high-level and low-level features is presented to extract more robust keypoints from optical and SAR images. Second, we use a two-channel network structure to improve the image patch matching performance. Benefiting from this design, the proposed method can directly learn the similarity between optical and SAR image patches without manually designing features and descriptors. Finally, the MKD and two-channel net-work are trained separately on GL3D and QXS-SAROPT data sets, and the PM-Net is tested on multiple pairs of optical and SAR images. The experimental results demonstrate that the proposed method outperforms four advanced image matching networks on qualitative and quantitative assessments. The quantitative experiment results show that using our method correct matching points numbers are increased by more than 1.15 times, the value of F1-measure is raised by an average of 7.4% and the root mean squared error (RMSE) is reduced by more than 15.3%. The advantages of MKD and the two-channel network are also verified through ablation experiments.

L1B+: A PERFECT SENSOR LOCALIZATION MODEL FOR SIMPLE SATELLITE STEREO RECONSTRUCTION FROM PUSH-FRAME IMAGE STRIPS

Abstract. We propose a novel method to generate a single image product from a multi-image strip acquired by a push-frame satellite imaging system. The images of the push-frame strips are combined into a large scale mosaic simulating a perfect sensor geometry. The local camera models of the input images are leveraged to produce a new localization model that covers the output mosaic entirely. Among other applications, this simplifies the task of stereo reconstruction enormously: instead of treating multiple stereo pairs of small images, it is possible to reconstruct the entire area covered by the push-frame acquisition using a single pair of mosaics incorporating all the images. We test our method using strips of SkySat L1B scenes and denote the output images as L1B+. To evaluate the quality of the L1B+ images and their localization models, the stereo reconstructions obtained with L1B+ are compared with those obtained with L1B and with a lidar reference model.

Intra- and Inter-Pair Consistency for Semi-Supervised Gland Segmentation.

Accurate gland segmentation in histology tissue images is a critical but challenging task. Although deep models have demonstrated superior performance in medical image segmentation, they commonly require a large amount of annotated data, which are hard to obtain due to the extensive labor costs and expertise required. In this paper, we propose an intra- and inter-pair consistency-based semi-supervised (I2CS) model that can be trained on both labeled and unlabeled histology images for gland segmentation. Considering that each image contains glands and hence different images could potentially share consistent semantics in the feature space, we introduce a novel intra- and inter-pair consistency module to explore such consistency for learning with unlabeled data. It first characterizes the pixel-level relation between a pair of images in the feature space to create an attention map that highlights the regions with the same semantics but on different images. Then, it imposes a consistency constraint on the attention maps obtained from multiple image pairs, and thus filters low-confidence attention regions to generate refined attention maps that are then merged with original features to improve their representation ability. In addition, we also design an object-level loss to address the issues caused by touching glands. We evaluated our model against several recent gland segmentation methods and three typical semi-supervised methods on the GlaS and CRAG datasets. Our results not only demonstrate the effectiveness of the proposed due consistency module and Obj-Dice loss, but also indicate that the proposed I2CS model achieves state-of-the-art gland segmentation performance on both benchmarks.

A Modified Approach of Extracting Landfast Ice Edge Based on Sentinel-1A InSAR Coherence Image in the Gulf of Bothnia

Landfast ice is an integral component of the coastal ecosystem. Extracting the edge and mapping the extent of landfast ice are one of the main methods for studying ice changes. In this work, a standardized process for extracting landfast ice edge in the Baltic Sea using the InSAR coherence image is established with Sentinel-1 radar data and InSAR technology. A modified approach combining multiscale segmentation and morphological erosion is then proposed to provide a reliable way to extract landfast ice edge. Firstly, the coherence image is obtained using InSAR technology. Then, the edge is separated and extracted with the modified approach. The modified approach is essentially a four-step procedure involving image segmentation, median filter, morphological erosion, and rejection of small patches. Finally, the full extent of landfast ice can be obtained using floodfill algorithm. Multiple InSAR image pairs of Sentinel-1A acquired from 2018 to 2019 are utilized to successfully extract the landfast ice edge in the Gulf of Bothnia. The results show that the landfast ice edge and the extents obtained by the proposed approach are visually consistent with those shown in the ice chart issued by the Swedish Meteorological and Hydrological Institute (SMHI) over a coastline length of 345 km. The mean distance between land–water boundary and the coastline issued by the National Oceanic and Atmospheric Administration (NOAA) is 109.1 m. The modified approach obviously preserves more details in local edge than the reference method. The experimental results show that the modified approach proposed in this paper can extract the edge and map the extent of landfast ice more accurately and quickly, and is therefore expected to contribute to the further understanding and analyzing the changes of landfast ice in the future.

Modality Translation in Remote Sensing Time Series

Modality translation, which aims to translate images from a source modality to a target one, has attracted a growing interest in the field of remote sensing recently. Compared to translation problems in multimedia applications, modality translation in remote sensing often suffers from inherent ambiguities, i.e., a single input image could correspond to multiple possible outputs, and the results may not be valid in the following image interpretation tasks, such as classification and change detection. To address these issues, we make the attempt to utilizing time-series data to resolve the ambiguities. We propose a novel multimodality image translation framework, which exploits temporal information from two aspects: 1) by introducing a guidance image from given temporally neighboring images in the target modality, we employ a feature mask module and transfer semantic information from temporal images to the output without requiring the use of any semantic labels and 2) while incorporating multiple pairs of images in time series, a temporal constraint is formulated during the learning process in order to guarantee the uniqueness of the prediction result. We also build a multimodal and multitemporal dataset that contains synthetic aperture radar (SAR), visible, and short-wave length infrared band (SWIR) image time series of the same scene to encourage and promote research on modality translation in remote sensing. Experiments are conducted on the dataset for two cross-modality translation tasks (SAR to visible and visible to SWIR). Both qualitative and quantitative results demonstrate the effectiveness and superiority of the proposed model.

A Coaxial Alignment Method for Large Flange Parts Assembly Using Multiple Local Images

The large flange parts in aero-engines are usually manually assembled. Collision damage caused by assembly alignment error often occurs in the assembly process, which affects the assembly accuracy and product reliability. The machine vision-based alignment methods usually achieve the high-precision measurement of the parts by obtaining the high-resolution images of the whole parts with a combination of laser distance sensors. Hence, existing methods are high costly and inefficient. In this paper, a new alignment method based on the principle of coaxial alignment for large flange parts is proposed. The proposed method first obtains multiple high-precision image pairs from the local field of views at the fitting surfaces of flanges. The clearance and bolt holes in each image pair are then extracted via edge recognition and Hough Transformation. Two optimization models are built to calculate the translation adjustment and rotation adjustment. The optimization model 1 is built with the translation adjustments of the flange as the variables and the consistency of the clearances as the objective function. The algorithm to solve the model based on the gradient-descent method is proposed. The positions of the bolt holes in the images are subsequently adjusted based on the translation adjustment, and the rotation adjustment is calculated by solving the built optimization model 2. The experiments show that the proposed method can be applied to the assembly process of large flange parts, and the visual servo control model based on this method also has good stability.

Three‐dimensional face modeling technology based on 5G virtual reality binocular stereo vision

Summary5G virtual reality technology is a technology that has attracted widespread attention in the computer field in recent years. Establishing a harmonious human–computer interaction environment in the multidimensional information space is one of the development goals of information technology. Face information is important information for identifying the identity of others. Three‐dimensional information has important applications in many fields and directions of machine vision. Therefore, the three‐dimensional common face image has a significant meaning. In this paper, binocular stereo vision system is used to collect multiple pairs of facial information images. The image correction method was used to correct the collected image pairs. According to the basic criteria of stereo matching, the advantages and disadvantages of various traditional image matching methods are analyzed. On the basis of completing the image correction, the graph cut matching method is selected as the matching method, and multiple pairs of collected image pairs are matched by this method to obtain a multi‐angle face disparity map. In this paper, after studying the basic dense matching algorithm and its improved algorithm, the SAD, SD, and NC methods of fixed window are used to compare the matching cost and the performance of the adaptive weight method in face matching. This paper studies the virtual roaming technology and the basis of the related Internet of Things (IoT) 3D face graphics theory and develops a building virtual roaming system based on MFC and OpenGL. The system draws on the basic I/0 device of the computer and provides users with a friendly MFC interface and multiple viewing angles and multiple roaming mode options for virtual faces. Experimental research shows that the reconstruction accuracy of the spatial points of the reconstructed face and the actual situation are both about 1%, which shows that the IoT 3D face modeling algorithm designed in this paper has high accuracy.

Virtual image pair-based spatio-temporal fusion

Spatio-temporal fusion is a technique used to produce images with both fine spatial and temporal resolution. Generally, the principle of existing spatio-temporal fusion methods can be characterized by a unified framework of prediction based on two parts: (i) the known fine spatial resolution images (e.g., Landsat images), and (ii) the fine spatial resolution increment predicted from the available coarse spatial resolution increment (i.e., a downscaling process), that is, the difference between the coarse spatial resolution images (e.g., MODIS images) acquired at the known and prediction times. Owing to seasonal changes and land cover changes, there always exist large differences between images acquired at different times, resulting in a large increment and, further, great uncertainty in downscaling. In this paper, a virtual image pair-based spatio-temporal fusion (VIPSTF) approach was proposed to deal with this problem. VIPSTF is based on the concept of a virtual image pair (VIP), which is produced based on the available, known MODIS-Landsat image pairs. We demonstrate theoretically that compared to the known image pairs, the VIP is closer to the data at the prediction time. The VIP can capture more fine spatial resolution information directly from known images and reduce the challenge in downscaling. VIPSTF is a flexible framework suitable for existing spatial weighting- and spatial unmixing-based methods, and two versions VIPSTF-SW and VIPSTF-SU are, thus, developed. Experimental results on a heterogeneous site and a site experiencing land cover type changes show that both spatial weighting- and spatial unmixing-based methods can be enhanced by VIPSTF, and the advantage is particularly noticeable when the observed image pairs are temporally far from the prediction time. Moreover, VIPSTF is free of the need for image pair selection and robust to the use of multiple image pairs. VIPSTF is also computationally faster than the original methods when using multiple image pairs. The concept of VIP provides a new insight to enhance spatio-temporal fusion by making fuller use of the observed image pairs and reducing the uncertainty of estimating the fine spatial resolution increment.

Lensed quasar search via time variability with the HSC transient survey

Gravitationally lensed quasars are useful for studying astrophysics and cosmology, and enlarging the sample size of lensed quasars is important for multiple studies. In this work, we develop a lens search algorithm for four-image (quad) lensed quasars based on their time variability. In the development of the lens search algorithm, we constructed a pipeline simulating multi-epoch images of lensed quasars in cadenced surveys, accounting for quasar variabilities, quasar hosts, lens galaxies, and the point spread function variation. Applying the simulation pipeline to the Hyper Suprime-Cam (HSC) transient survey, an ongoing cadenced survey, we generated HSC-like difference images of the mock lensed quasars from the lens catalog of Oguri & Marshall (2010, MNRAS, 405, 2579). With the difference images of the mock lensed quasars and the variable objects from the HSC transient survey, we developed a lens search algorithm that picks out variable objects as lensed quasar candidates based on their spatial extent in the difference images. We tested the performance of our lens search algorithm on a sample combining the mock lensed quasars and variable objects from the HSC transient survey. Using difference images from multiple epochs, our lens search algorithm achieves a high true-positive rate (TPR) of 90.1% and a low false-positive rate (FPR) of 2.3% for the bright quads (the third brightest image brightness m3rd < 22.0 mag) with wide separation (the largest separation among the multiple image pairs θLP > 1.5″). With a preselection of the number of blobs in the difference image, we obtain a TPR of 97.6% and a FPR of 2.6% for the bright quads with wide separation. Even when difference images are only available in one single epoch, our lens search algorithm can still detect the bright quads with wide separation at high TPR of 97.6% and low FPR of 2.4% in the optimal seeing scenario, and at TPR of ∼94% and FPR of ∼5% in typical scenarios. Therefore, our lens search algorithm is promising and is applicable to ongoing and upcoming cadenced surveys, particularly the HSC transient survey and the Rubin Observatory Legacy Survey of Space and Time, for finding new lensed quasar systems.

3D target localization based on multi–unmanned aerial vehicle cooperation

Ground target three-dimensional positions measured from optical remote-sensing images taken by an unmanned aerial vehicle play an important role in related military and civil applications. The weakness of this system lies in its localization accuracy being unstable and its efficiency being low when using a single unmanned aerial vehicle. In this paper, a novel multi–unmanned aerial vehicle cooperative target localization measurement method is proposed to overcome these issues. In the target localization measurement stage, three or more unmanned aerial vehicles simultaneously observe the same ground target and acquire multiple remote-sensing images. According to the principle of perspective projection, the target point, its image point, and the camera’s optic center are collinear, and nonlinear observation equations are established. These equations are then converted to linear equations using a Taylor expansion. Robust weighted least-squares estimation is used to solve the equations with the objective function of minimizing the weighted square sum of re-projection errors from target points to multiple pairs of images, which can make the best use of the effective information and avoid interference from the observation data. An automatic calculation strategy using a weight matrix is designed, and the weight matrix and target-position coordinate value are updated in each iteration until the iteration stopping condition is satisfied. Compared with the stereo-image-pair cross-target localization method, the multi–unmanned aerial vehicle cooperative target localization method can use more observation information, which results in higher rendezvous accuracy and improved performance. Finally, the effectiveness and robustness of this method is verified by numerical simulation and flight testing. The results show that the proposed method can effectively improve the precision of the target’s localization and demonstrates great potential for providing more accurate target localization in engineering applications.

Three-dimensional reconstruction of the dribble track of soccer robot based on heterogeneous binocular vision

With the rapid development of science and technology, as an important branch of the computer. Binocular vision technology has also been greatly improved and improved. It has been applied to many fields of production and life, medical information, industrial manufacturing, and service industries. Heterogeneous binocular vision typically uses a camera to simulate the sensory acquisition process of humans. To process the target information, multiple viewpoints observe the same motion scene. Finally, multiple pairs of images are obtained from different viewpoints. Three-dimensional reconstruction has long been one of the important research fields of computer vision. It has great application value in robot autonomous navigation vision, self-driving vehicle, and automatic detection field and freedom degree mechanical device control. In the robot application of visual positioning technology, with the development needs, the level of intelligence of the robot is also gradually increasing. The three-dimensional reconstruction technology combined with the principle of heterogeneous binocular vision has important practical significance for completing the technique of the dribble track of soccer robots. Based on the RoboCup soccer robot competition, this paper uses the heterogeneous binocular vision technology to analyze the kinematics of the soccer robot in detail. By analyzing the basic motion characteristics of soccer robots, a three-dimensional reconstruction technology model of its dribble trajectory is established, and the reliability, practicability and rationality of the design are verified.

Multiple-image encryption and hiding with an optical diffractive neural network

A cascaded phase-only mask architecture (or an optical diffractive neural network) can be employed for different optical information processing tasks such as pattern recognition, orbital angular momentum (OAM) mode conversion, image salience detection and image encryption. However, for optical encryption and watermarking applications, such a system usually cannot process multiple pairs of input images and output images simultaneously. In our proposed scheme, multiple input images can be simultaneously fed to an optical diffractive neural network (DNN) system and each corresponding output image will be displayed in a non-overlap sub-region in the output imaging plane. Each input image undergoes a different optical transform in an independent channel within the same system. The multiple cascaded phase masks in the system can be effectively optimized by a wavefront matching algorithm. Similar to recent optical pattern recognition and mode conversion works, the orthogonality property is employed to design a multiplexed DNN.

Super-resolution reconstruction of remote sensing images based on convolutional neural network

A method of super-resolution reconstruction of remote sensing images based on convolutional neural network is proposed to address the problems of low-resolution and poor visual quality of remote sensing images. In this method, a sample database with high-resolution (HR) and low-resolution (LR) remote sensing images was constructed. A convolutional neural network was then obtained by determining the intrinsic relationship between HR and LR remote sensing images in the sample database. Multiple pairs of HR and LR images were selected from the sample database and sent into a six-layer convolutional neural network. The experimental results showed that compared with other learning-based methods, such as the fast super-resolution convolutional neural network (FSRCNN), the image quality obtained by our method is improved both subjectively and objectively. Moreover, the training time was ∼17 % less than in the FSRCNN method.

Multi-Image Robust Alignment of Medium-Resolution Satellite Imagery

This paper describes an automatic multi-image robust alignment (MIRA) procedure able to simultaneously co-register a time series of medium-resolution satellite images in a bundle block adjustment (BBA) fashion. Instead of the direct co-registration of each image with respect to a reference ‘master’ image on the basis of corresponding features, MIRA also considers those tie points that may be not be shared with the master, but they only connect the other images (‘slaves’) among them. In a first stage, tie points are automatically extracted by using pairwise feature-based matching based on the SURF operator. In a second stage, such extracted features are re-ordered to find corresponding tie points visible on multiple image pairs. A ‘master’ image is then selected with the only purpose to establish the datum of the final image alignment and to instantiate the computation of approximate registration parameters. All the available information obtained so far is fed into a least-squares BBA to estimate the unknowns, which include the registration parameters and the coordinates of tie points re-projected in the ‘master’ image space. The analysis of inner and outer reliability of the observations is applied to assess whether the residual blunders may be located using data snooping, and to evaluate the influence of undetected outliers on the final registration results. Three experiments with simulated datasets and one example consisting of eleven Landsat-5/TM images are reported and discussed. In the case of real data, results have been positively checked against the ones obtained by using alternative procedures (BBA with manual measurements and ‘slave-to-master’ registration based on automatically extracted tie points). These experiments have confirmed the correctness of the MIRA approach and have highlighted the potential of the inner control on the final quality of the solution that may come from the reliability analysis.

Cardiac Motion Correction for Helical CT Scan With an Ordinary Pitch.

Cardiac X-ray computed tomography (CT) imaging is still challenging due to the cardiac motion during CT scanning, which leads to the presence of motion artifacts in the reconstructed image. In response, many cardiac X-ray CT imaging algorithms have been proposed, based on motion estimation (ME) and motion compensation (MC), to improve the image quality by alleviating the motion artifacts in the reconstructed image. However, these ME/MC algorithms are mainly based on an axial scan or a low-pitch helical scan. In this paper, we propose a ME/MC-based cardiac imaging algorithm for the data set acquired from a helical scan with an ordinary pitch of around 1.0 so as to obtain the whole cardiac image within a single scan of short time without ECG gating. In the proposed algorithm, a sequence of partial angle reconstructed (PAR) images is generated by using consecutive parts of the sinogram, each of which has a small angular span. Subsequently, an initial 4-D motion vector field (MVF) is obtained using multiple pairs of conjugate PAR images. The 4-D MVF is then refined based on an image quality metric so as to improve the quality of the motion-compensated image. Finally, a time-resolved cardiac image is obtained by performing motion-compensated image reconstruction by using the refined 4-D MVF. Using digital XCAT phantom data sets and a human data set commonly obtained via a helical scan with a pitch of 1.0, we demonstrate that the proposed algorithm significantly improves the image quality by alleviating motion artifacts.

Unsupervised learning of complex associations in an animal model

Supervised learning results from explicit corrective feedback, whereas unsupervised learning results from statistical co-occurrence. In an initial training phase, we gave pigeons an unsupervised learning task to see if mere pairing could establish associations between multiple pairs of visual images. To assess learning, we administered occasional testing trials in which pigeons were shown an object and had to choose between previously paired and unpaired tokens. Learning was evidenced by preferential choice of the previously unpaired token. In a subsequent supervised training phase, learning was facilitated if the object and token had previously been paired. These results document unsupervised learning in pigeons and resemble statistical learning in infants, suggesting an important parallel between human and animal cognition.

SAR-SIFT for Matching Multiple SAR Images and Radargrammetry

This article presents a new application of the SAR-SIFT algorithm proposed by Dellinger et al. (IEEE Trans Geosci Remote Sens 53:453–466, 2015) for the automatic generation of tie points (TPs). In particular, SAR-SIFT is applied on stereo-SAR images to extract corresponding points and determine their 3D position. Furthermore, the potential of the combined use of the SAR-SIFT with the SAR-stereo model for extracting reliable TPs is analysed, by evaluating their absolute accuracy in 2D and 3D. The tests are performed with TerraSAR-X HR SpotLight data acquired over Trento, Italy. Single and multiple image pairs have been considered. 3D accuracy assessment of the extracted points is performed relying on LiDAR data; the results show a planimetric accuracy up to 1 m and an elevation accuracy of about 2 m.

Development and Validation of a Photonumeric Scale for Assessment of Chin Retrusion.

BACKGROUNDA validated scale is needed for objective and reproducible comparisons of chin appearance before and after chin augmentation in practice and clinical studies.OBJECTIVETo describe the development and validation of the 5-point photonumeric Allergan Chin Retrusion Scale.METHODSThe Allergan Chin Retrusion Scale was developed to include an assessment guide, verbal descriptors, morphed images, and real subject images for each scale grade. The clinical significance of a 1-point score difference was evaluated in a review of multiple image pairs representing varying differences in severity. Interrater and intrarater reliability was evaluated in a live-subject validation study (N = 298) completed during 2 sessions occurring 3 weeks apart.RESULTSA difference of ≥1 point on the scale was shown to reflect a clinically meaningful difference (mean [95% confidence interval] absolute score difference, 1.07 [0.94–1.20] for clinically different image pairs and 0.51 [0.39–0.63] for not clinically different pairs). Intrarater agreement between the 2 live-subject validation sessions was substantial (mean weighted kappa = 0.79). Interrater agreement was substantial during the second rating session (0.68, primary end point).CONCLUSIONThe Allergan Chin Retrusion Scale is a validated and reliable scale for physician rating of severity of chin retrusion.