Reference Image Selection Research Articles

Automated image registration of RGB, hyperspectral and chlorophyll fluorescence imaging data

BackgroundThe early and specific detection of abiotic and biotic stresses, particularly their combinations, is a major challenge for maintaining and increasing plant productivity in sustainable agriculture under changing environmental conditions. Optical imaging techniques enable cost-efficient and non-destructive quantification of plant stress states. Monomodal detection of certain stressors is usually based on non-specific/indirect features and therefore is commonly limited in their cross-specificity to other stressors. The fusion of multi-domain sensor systems can provide more potentially discriminative features for machine learning models and potentially provide synergistic information to increase cross-specificity in plant disease detection when image data are fused at the pixel level.ResultsIn this study, we demonstrate successful multi-modal image registration of RGB, hyperspectral (HSI) and chlorophyll fluorescence (ChlF) kinetics data at the pixel level for high-throughput phenotyping of A. thaliana grown in Multi-well plates and an assay with detached leaf discs of Rosa × hybrida inoculated with the black spot disease-inducing fungus Diplocarpon rosae. Here, we showcase the effects of (i) selection of reference image selection, (ii) different registrations methods and (iii) frame selection on the performance of image registration via affine transform. In addition, we developed a combined approach for registration methods through NCC-based selection for each file, resulting in a robust and accurate approach that sacrifices computational time. Since image data encompass multiple objects, the initial coarse image registration using a global transformation matrix exhibited heterogeneity across different image regions. By employing an additional fine registration on the object-separated image data, we achieved a high overlap ratio. Specifically, for the A. thaliana test set, the overlap ratios (ORConvex) were 98.0 ± 2.3% for RGB-to-ChlF and 96.6 ± 4.2% for HSI-to-ChlF. For the Rosa × hybrida test set, the values were 98.9 ± 0.5% for RGB-to-ChlF and 98.3 ± 1.3% for HSI-to-ChlF.ConclusionThe presented multi-modal imaging pipeline enables high-throughput, high-dimensional phenotyping of different plant species with respect to various biotic or abiotic stressors. This paves the way for in-depth studies investigating the correlative relationships of the multi-domain data or the performance enhancement of machine learning models via multi modal image fusion.

A novel approach for color-balanced reference image selection for breast histology image normalization

Histology image analysis plays a crucial role in pathological research and diagnosis. The effectiveness of Computer-Aided Detection procedures largely depends on the normalization of histology images to ensure precise diagnoses. On the other hand, the success of this normalization process is tied to the choice of an appropriate reference image, which acts as a standard for aligning and modifying the color and intensity of other images in the dataset. However, the lack of a universally reliable method for selecting an appropriate reference image remains a substantial obstacle in histology image analysis. In this direction, the study introduces an automated procedure for selecting a reference image that is designed to accurately represent the entire spectrum of the dataset or its specific class under investigation. The impact of the proposed method on the morphological and structural quality of normalized histology images is investigated using three established normalization techniques: Reinhard, Macenko, and Vahadane. Moreover, the study also utilized two strategies: the global reference image strategy, where a single reference image is chosen for the entire dataset, and the class-specific reference image strategy, which considers the unique characteristics of each class. The results demonstrate that the proposed automatic reference image selection procedure consistently outperforms random selection in both strategies, yielding higher metric scores (SSIM, QSSIM, and PCC). Furthermore, the class-specific reference image approach significantly improves the image quality compared to random selection. The proposed method enhances alignment and accuracy in histology image normalization by leveraging class-specific reference images, facilitating more reliable analysis in histopathology research and clinical applications.

Hierarchical lossless coding of light fields with improved random access

The adoption of light field (LF) technology is often hindered by the large amount of data that is necessary to represent such information. To alleviate such burden, specific coding techniques have been under research in recent years. However, most of them require decoding the whole LF data, even in applications where only a region of the LF visual content needs to be accessed. Therefore, encoding schemes that incorporate scalability and random access capabilities would cater to a wider range of LF use cases, particularly those requiring lossless compression. To enable such type of functionalities, an encoder must be designed so that the decoding process can extract only specific views/regions of the complete LF, according to user options or application requirements. This paper proposes hierarchical Minimum Rate Predictors (H-MRP), a lossless light field codec that provides the aforementioned functionalities. The main contributions herein proposed are: (i) the lossless encoding the LF in multiple hierarchical layers; (ii) the flexible selection of reference images for prediction; and (iii) the use disparity compensation between the reference images. The experimental results show that the proposed encoder provides a good compromise in terms of compression efficiency versus random access penalty, surpassing its best competitor by 25% in terms of random access and by 1 bit-per-pixel in terms of compression efficiency. When optimised to provide the best compression efficiency, the method surpasses current state-of-the-art lossless encoding by up to 3 bits-per-pixel.

A Flexible Reference-Insensitive Spatiotemporal Fusion Model for Remote Sensing Images Using Conditional Generative Adversarial Network

Due to the tradeoff between spatial and temporal resolutions of remote sensing images, spatiotemporal fusion models were proposed to synthesize the high spatiotemporal image series. Currently, spatiotemporal fusion models usually employ one coarse-resolution image acquired on a prediction date and at least another pair of coarse–fine resolution images close to the prediction time as references to derive the fine-resolution image on the prediction date. After years of development, the model accuracy has gained a certain improvement, but nearly, all the models require at least three image inputs and rigid time constraints must be applied to the references to guarantee the fusion accuracy. However, it is not always that easy to collect adequate data pairs for fine-resolution image series simulation in practice because of the bad weather condition or the time inconsistency between the coarse–fine resolution data sources, which causes some difficulties in the actual application. This article introduces the conditional generative adversarial network (CGAN) and switchable normalization technique into the spatiotemporal fusion problem and proposes a flexible deep network named the GAN-based SpatioTemporal Fusion Model (GAN-STFM) to reduce the number of model inputs and broke the time restriction on reference image selection. The GAN-STFM just needs a coarse-resolution image on the prediction date and another fine-resolution reference image at an arbitrary time in the same area for model inputs. As far as we know, this is the first spatiotemporal fusion model that requires only two images as model inputs and puts no restriction on the acquisition time of references. Even so, the GAN-STFM performs on par or better than other classical fusion models in the experiments. With this improvement, the data preparation for spatiotemporal fusion tends to be much easier than before, showing a promising perspective for practical applications.

Method of reference image selection to provide high-speed aircraft navigation under conditions of rapid change of flight trajectory

Method of reference image selection to provide high-speed aircraft navigation under conditions of rapid change of flight trajectory

Few-shot learning for deformable image registration in 4DCT images.

To develop a rapid and accurate 4D deformable image registration (DIR) approach for online adaptive radiotherapy. We propose a deep learning (DL)-based few-shot registration network (FR-Net) to generate deformation vector fields from each respiratory phase to an implicit reference image, thereby mitigating the bias introduced by the selection of reference images. The proposed FR-Net is pretrained with limited unlabeled 4D data and further optimized by maximizing the intensity similarity of one specific four-dimensional computed tomography (4DCT) scan. Because of the learning ability of DL models, the few-shot learning strategy facilitates the generalization of the model to other 4D data sets and the acceleration of the optimization process. The proposed FR-Net is evaluated for 4D groupwise and 3D pairwise registration on thoracic 4DCT data sets DIR-Lab and POPI. FR-Net displays an averaged target registration error of 1.48 mm and 1.16 mm between the maximum inhalation and exhalation phases in the 4DCT of DIR-Lab and POPI, respectively, with approximately 2 min required to optimize one 4DCT. Overall, FR-Net outperforms state-of-the-art methods in terms of registration accuracy and exhibits a low computational time. We develop a few-shot groupwise DIR algorithm for 4DCT images. The promising registration performance and computational efficiency demonstrate the prospective applications of this approach in registration tasks for online adaptive radiotherapy. This work exploits DL models to solve the optimization problem in registering 4DCT scans while combining groupwise registration and few-shot learning strategy to solve the problem of consuming computational time and inferior registration accuracy.

A multi-scale patch-wise algorithm for multi-exposure image fusion

Multi-exposure fusion is a high dynamic range image generation method. The challenge it faces is to reconstruct artifact-free high dynamic range images for dynamic scenes containing moving objects. This paper proposes a structural patch decomposition algorithm based on the Laplacian pyramid to obtain high dynamic range images without artifacts. This paper discusses the relationship between the total exposure quality score and the number of poorly exposed pixels and proposes a robust reference image selection method. It avoided the local exposure distortion in the fused image caused by the improper selection of reference images and improved the algorithm’s fusion effect in dynamic scenes. More importantly, this paper inject a new fusion mechanism into the conventional Laplacian pyramid fusion framework, namely the non-standardized structural patch decomposition algorithm. While retaining its original advantages, it reduces the computational complexity of the algorithm. Experimental results on dynamic and static scenes show that the proposed algorithm can generate fused images with a clear structure and almost no halo artifacts.

A large-scale 2005–2012 flood map record derived from ENVISAT-ASAR data: United Kingdom as a test case

Synthetic Aperture Radars (SAR) are adequate sensors for mapping water bodies from space as they can be used to acquire data of equal quality both day and night and practically regardless of weather conditions. Furthermore, the global coverage of SAR data provides an opportunity to generate global scale flood records that are essential for improving our understanding of flood risks worldwide and of how these risks are changing over time. In this study, we introduce an automatic change-detection based method that allows global-scale flood records to be generated using the readily and freely available ENVISAT-ASAR data collection. It consists of the following three steps: (i) flood image identification; (ii) reference image selection; (iii) floodwater detection. As a test case, this study uses all available ENVISAT-ASAR images from eight different orbital tracks that were acquired over the United Kingdom over the period 2005–2012. Due to a lack of large-scale ground truth data, the evaluation of the results is carried out using different data sources. First, subsets of the flood maps over the Severn River basin are evaluated using a flood extent map that was manually digitized from very high-resolution aerial imagery. According to our results, the overall accuracy of both flood maps' subsets is higher than 85% while the user accuracy of the flood class is above 88%. Next, for the regions and images without available ground truth data, a visual inspection is carried out using simulations generated by the hydraulic model LISFLOOD-FP, as well as LANDSAT 7 ETM+ images obtained with a 30 m spatial resolution. Meanwhile, by comparing the acquisition dates of identified flood SAR images, the LISFLOOD-FP model results and optical data, a good agreement has been found. The experimental results over the United Kingdom indicate that the proposed method has strong potential for the generation of a global flood data record from the ENVISAT-ASAR archive.

Intelligent Reference Curation for Visual Place Recognition Via Bayesian Selective Fusion

A key challenge in visual place recognition (VPR) is recognizing places despite drastic visual appearance changes due to factors such as time of day, season, weather or lighting conditions. Numerous approaches based on deep-learnt image descriptors, sequence matching, domain translation, and probabilistic localization have had success in addressing this challenge, but most rely on the availability of carefully curated representative reference images of the possible places. In this letter, we propose a novel approach, dubbed Bayesian Selective Fusion, for actively selecting and fusing informative reference images to determine the best place match for a given query image. The selective element of our approach avoids the counterproductive fusion of every reference image and enables the dynamic selection of informative reference images in environments with changing visual conditions (such as indoors with flickering lights, outdoors during sunshowers or over the day-night cycle). The probabilistic element of our approach provides a means of fusing multiple reference images that accounts for their varying uncertainty via a novel training-free likelihood function for VPR. On difficult query images from two benchmark datasets, we demonstrate that our approach matches and exceeds the performance of several alternative fusion approaches along with state-of-the-art techniques that are provided with prior (unfair) knowledge of the best reference images. Our approach is well suited for long-term robot autonomy where dynamic visual environments are commonplace since it is training-free, descriptor-agnostic, and complements existing techniques such as sequence matching.

Performance assessment of two motion management systems for frameless stereotactic radiosurgery

Background/PurposeFrameless stereotactic radiosurgery (SRS) requires dedicated systems to monitor patient motion in order to avoid inaccurate radiation delivery due to involuntary shifts. The purpose of this study is to assess the accuracy and sensitivity of two distinct motion monitoring systems used for frameless SRS.MethodsA surface image-guided system known as optical surface monitoring system (OSMS), and a fiducial marker-based system known as high definition motion management (HDMM) as part of the latest Gamma Knife Icon® were compared. A 3D printer-based cranial motion phantom was developed to evaluate the accuracy and sensitivity of these two systems in terms of: (1) the capability to recognize predefined shifts up to 3 cm, and (2) the capability to recognize predefined speeds up to 3 cm/s. The performance of OSMS, in terms of different reference surfaces, was also evaluated.ResultsTranslational motion could be accurately detected by both systems, with an accuracy of 0.3 mm for displacement up to 1 cm, and 0.5 mm for larger displacements. The reference surface selection had an impact on OSMS performance, with flat surface resulting in less accuracy. HDMM was in general more sensitive when compared with OSMS in capturing the motion, due to its faster frame rate, but a delay in response was observed with faster speeds. Both systems were less sensitive in detection of superior-inferior motion when compared to lateral or vertical displacement directions.ConclusionTranslational motion can be accurately and sensitively detected by OSMS and HDMM real-time monitoring systems. However, performance variations were observed along different motion directions, as well as amongst the selection of reference images. Caution is needed when using real-time monitoring systems for frameless SRS treatment.

A privacy-preserving protocol for efficient nighttime haze removal using cloud based automatic reference image selection and color transfer as a service

The advanced Internet technologies have migrated the people to rejoice a virtual environment known as cloud computing. The user can avail the desired services on a pay-as-you-go model, without worrying about the burden of infrastructure maintenance. However, privacy is one of the major issues in cloud computing. This issue is further widened for highly confidential multimedia data like surveillance images and videos. In the context of cloud based smart multimedia systems, it has been found that due to inconsistent weather conditions, there is a usual requirement of post-processing the captured multimedia for better appearance. However, privacy related concerns are resisting users to move their data to the cloud. One such problem is addressed in this paper, specializing the task of efficient nighttime haze removal using privacy-preserving cloud based automatic reference image selection and color transfer as a service. Different from daytime conditions, nighttime haze image consists of multiple light sources, which makes an ambiguous situation for haze removal. We address this problem by first selecting an appropriate gray image as the reference and then transferring its colors to nighttime haze image. This makes the transformed image a suitable candidate for radiance recovery. The proposed protocol is designed to securely outsource this considerable burden from user end. We accomplish this by first proposing an automatic reference gray image selection method, followed by efficient handling mechanisms for technical challenges arising due to performing color transfer operations securely over cloud. Experimental results and validation demonstrates superiority of the proposed method over state-of-the-art schemes. Security analysis of the proposed protocol is established through a challenge-response game model.

A luminance compensation method for fisheye video panorama stitching

The fisheye camera has becoming the most popular panoramic video capturing device. In fact, the stitching for fisheye images has drawn more and more attention. To solve the artifacts resulting from the exposure difference of several cameras, this paper proposes a luminance compensation method based on an uneven sampling histogram. According to the transition of the spatial sampling rate with the latitudes increasing in panorama, the proposed method samples the overlapped region of two adjacent images to calculate the histograms, and matches the histograms one by one to balance the luminance. Moreover, in consideration of the influence of luminance distribution to panorama video quality, we present a method for adaptive reference image selection. The selected reference image based on the proposed method can improve the overall quality of the panorama effectively. Due to the consideration of the distortion characteristics of fisheye images, the proposed method does not require an accurate alignment, and it is also well-adapted to luminance variations. Experimental results show that the proposed method can reduce the time complexity for panorama video stitching significantly without obvious quality degeneration, and it is suitable for fisheye lens-based real-time panoramic video stitching.

Novel Image Set Compression Algorithm Using Rate-Distortion Optimized Multiple Reference Image Selection

Image set compression has recently become an active research topic due to the explosion of digital photographs. In order to efficiently compress the image sets of similar images with moving objects, in this paper, we propose a novel algorithm for image set compression using multiple reference images. First, for an image set, its depth-constrained minimum arborescence is generated. We then present a reference image candidate determination method to build the reference image candidates for the images of the set. Furthermore, we propose a rate-distortion optimized multiple reference image selection method. This method compares the correlation between every image and each of its reference image candidates to produce its multiple reference images. Finally, compressed image data are achieved by employing block-based motion compensation and residue coding. In addition, we also give a new way of access to images to keep the same access delay with single reference image-based schemes. Compared with the state-of-the-art image compression algorithms, experimental results show that our proposed algorithm can significantly improve the image compression performance.

Colorization Using Neural Network Ensemble.

This paper investigates into the colorization problem, which converts a grayscale image to a colorful version. This is a difficult problem and normally requires manual adjustment to achieve artifact-free quality. For instance, it normally requires human-labeled color scribbles on the grayscale target image or a careful selection of colorful reference images. The recent learning-based colorization techniques automatically colorize a grayscale image using a single neural network. Since different scenes usually have distinct color styles, it is difficult to accurately capture the color characteristics using a single neural network. We propose a mixture learning model representing the presence of sub-color-style within an overall image data set. We, therefore, ensemble multiple neural networks to obtain better color estimation performance than could be obtained from any of the constituent neural network alone. A two-step colorization strategy is utilized as an adaptive color style clustering followed by a neural network ensemble. To ensure artifact-free quality, a joint bilateral filtering-based post-processing step is proposed. Numerous experiments demonstrate that our method generates high-quality results comparable with state-of-the-art algorithms.

Automatic Reference Image Selection for Color Balancing in Remote Sensing Imagery Mosaic

Selection of a reference image is an important step in color balancing. However, the past research and currently available methods do not focus on it, leading to the lack of an effective way to select the reference image for color balancing in remote sensing imagery mosaic. This letter proposes a novel automatic reference image selection method that aims to select the reference images by assessing multifactors according to the land surface types of the target images. The proposed method addresses the limitations caused by the use of a single assessment factor as well as the selection of a single image as the reference in traditional methods. In addition, the proposed method has a wider range of applications than those requiring no reference image. The visual experimental results indicate that the proposed method can select the suitable reference images, which benefits the color balancing result, and outperforms the other comparative methods. Moreover, the absolute mean value of skewness metric of the proposed method is 0.0831, which is lower than the values of the other comparison methods. It indicates that the result of the proposed method had the best performance in the color information. The quantitative analyses with the metric of absolute difference of mean value indicate that the proposed method has a good ability in maintaining the spectral information, and the spectral changing rates had been reduced at least 10.66% by the proposed method when compared with the other methods.

Direct Digital Surface Model Generation by Semi-Global Vertical Line Locus Matching

As the core issue for Digital Surface Model (DSM) generation, image matching is often implemented in photo space to get disparity or depth map. However, DSM is generated in object space with additional processes such as reference image selection, disparity maps fusion or depth maps merging, and interpolation. This difference between photo space and object space leads to process complexity and computation redundancy. We propose a direct DSM generation approach called the semi-global vertical line locus matching (SGVLL), to generate DSM with dense matching in the object space directly. First, we designed a cost function, robust to the pre-set elevation step and projection distortion, and detected occlusion during cost calculation to achieve a sound photo-consistency measurement. Then, we proposed an improved semi-global cost aggregation with guidance of true-orthophoto to obtain superior results at weak texture regions and slanted planes. The proposed method achieves performance very close to the state-of-the-art with less time consumption, which was experimentally evaluated and verified using nadir aerial images and reference data.

Mosaicking of Unmanned Aerial Vehicle Imagery in the Absence of Camera Poses

The mosaicking of Unmanned Aerial Vehicle (UAV) imagery usually requires information from additional sensors, such as Global Position System (GPS) and Inertial Measurement Unit (IMU), to facilitate direct orientation, or 3D reconstruction approaches (e.g., structure-from-motion) to recover the camera poses. In this paper, we propose a novel mosaicking method for UAV imagery in which neither direct nor indirect orientation procedures are required. Inspired by the embedded deformation model, a widely used non-rigid mesh deformation model, we present a novel objective function for image mosaicking. Firstly, we construct a feature correspondence energy term that minimizes the sum of the squared distances between matched feature pairs to align the images geometrically. Secondly, we model a regularization term that constrains the image transformation parameters directly by keeping all transformations as rigid as possible to avoid global distortion in the final mosaic. Experimental results presented herein demonstrate that the accuracy of our method is twice as high as an existing (purely image-based) approach, with the associated benefits of significantly faster processing times and improved robustness with respect to reference image selection.

TU-F-CAMPUS-J-05: Quantitative Evaluation of the Relationship Between Tissue Velocity and Motion-Artifacts of Free-Breathing Low-Dose Fast-Helical CT Scans

Purpose: The process of characterizing breathing motion has recently advanced by employing repeated fast-helical scans as the free-breathing CT images. While the scans are acquired using state-of-the-art CT scanners operating at nearly their highest couch-speeds, it still requires 0.23 s to scan any specific tissue-region. During quiet respiration, tissue velocities of up to 4 cm per second have been observed; leading to the residual motion-artifacts in the fast helical CT scans. This work evaluates the magnitude and characteristics of the motion-artifacts and correlates against model-calculated tissue velocities. Methods: 10 Patients were scanned 25 successive times using a low-dose fast-helical protocol with scans acquired in alternating directions. A 64-slice CT scanner with a pitch of 1.2 and table speed of 161.7 mm/s was used. The first scan was selected as the reference scan and deformable registrations were performed to register the other 24 scans to the reference scan. Motion-blur was quantified using blurring metrics and doubling-artifacts were further quantified using edge-response width to determine the mean edge-width of each lung diaphragm. The model-predicted velocities were correlated against the quantified artifacts. Results: Motion-artifacts appeared with the increased tissue velocity, from artifact-free to blurring-artifacts and then doubling-artifacts. Increasing amounts of doubling-artifacts were observed with tissue velocities > 18.2±1.3 mm/s. Regional blurring-artifacts occurred with velocities > 3.8±2.3 mm/s. The relationship between tissue velocity and motion-artifact severity was linear. There was no statistically significant difference in artifact magnitude between inhalation and exhalation phases. Conclusion: In spite of employing fast-helical acquisition with table speeds of 161.7 mm/s, motion-artifacts remained. The relationship between motion-artifact amplitude and tissue-velocity was linear with free-breathing fast-helical scans. This relationship will be used to aid in reference image selection, guide 5DCT accuracy assessments, and determine which images are averaged to produce the reference phase image used for treatment planning.

MO‐FG‐204‐02: Reference Image Selection in the Presence of Multiple Scan Realizations

Purpose:Fusing information from multiple correlated realizations (e.g., 4DCT) can improve image quality. This process often involves ill‐conditioned and asymmetric nonlinear registration and the proper selection of a reference image is important. This work proposes to examine post‐registration variation indirectly for such selection, and develops further insights to reduce the number of cross‐registrations needed.Methods:We consider each individual scan as a noisy point in the vicinity of an image manifold, related by motion. Nonrigid registration “transports” a scan along the manifold to the reference neighborhood, and the residual is a surrogate for local variation. To test this conjecture, 10 thoracic scans from the same session were reconstructed from a recently developed low‐dose helical 4DCT protocol. Pairwise registration was repeated bi‐directionally (81 times) and fusion was performed with each candidate reference. The fused image quality was assessed with SNR and CNR. Registration residuals in SSD, harmonic energy, and deformation Jacobian behavior were examined. The semi‐symmetry is further utilized to reduce the number of registration needed.Results:The comparison of image quality between single image and fused ones identified reduction of local intensity variance as the major contributor of image quality, boosting SNR and CNR by 5 to 7 folds. This observation further suggests the criticality of good agreement across post‐registration images. Triangle inequality on the SSD metric provides a proficient upper‐bound and surrogate on such disagreement. Empirical observation also confirms that fused images with high residual SSD have lower SNR and CNR than the ones with low or intermediate SSDs. Registration SSD is structurally close enough to symmetry for reduced computation.Conclusion:Registration residual is shown to be a good predictor of post‐fusion image quality and can be used to identify good reference centers. Semi‐symmetry of the registration residual further reduces computation cost.Supported by in part by NIH R01 CA096679.

Automatic selection of color reference image for panoramic stitching

Color correction is an important step in the generation of high-resolution stitched panoramas. Typically, color correction schemes try to match the color of each image in the panorama to an arbitra...