Mosaicking Methods Research Articles

A flexible multi-temporal orthoimage mosaicking method based on dynamic variable patches

Orthoimage mosaicking plays an important role in remote sensing applications, such as environment monitoring and protection. However, the cloud coverage in remote sensing images often results in substantial loss of information and degradation of data quality, bringing lots of challenges to orthoimage mosaicking. Therefore, this paper proposes a flexible multi-temporal orthoimage mosaicking method based on dynamic variable patches. Different from most existing pixel-based mosaicking methods, the proposed method defines patch as the minimal processing unit for orthoimage mosaicking, and the size and position of each patch are dynamic variable. Patches from different images are rearranged under the image texture consistency constraint to generate the mosaic image. In the proposed method, the initial positions of available patches of all images are determined first, then the initial size of each patch is determined by the cloud coverage and the effective area of image. Subsequently, considering the inevitable misalignments between different orthoimages, the texture similarity between patches is utilized to select the most suitable patch, and the size of the patch is also dynamically adjusted to make the misalignments within each patch controllable. Finally, considering the color inconsistency in the overlapping areas of neighboring images, an area-weighted image blending algorithm is also presented to achieve invisible seams and smooth transitions in the final mosaic. Not only does the proposed method produce mosaic images with minimal cloud coverage areas by excluding cloud pixels as much as possible, but it is also flexible for the mosaicking of images with varying degrees of misalignments, which is achieved by dynamically adjusting the size and position of each patch while maintaining texture consistency constraint. Both simulated experiments and real data experiments are carried out to verify the performance of the proposed method. The experimental results demonstrate that the proposed method can generate the mosaic image with higher spatial continuity and radiometric consistency.

Assessing the capability of Sentinel-2 time-series to estimate soil organic carbon and clay content at local scale in croplands

The use of remote sensing data methods is affordable for the mapping of soil properties of the plowed layer over croplands. Carried out in the framework of the ongoing STEROPES project of the European Joint H2020 Program SOIL, this work is focused on the feasibility of Sentinel-2 based approaches for the high resolution mapping of topsoil clay and organic carbon (SOC) contents at the within-farm or within-field scales, for cropland sites of contrasted climates and soil types across the Northern hemisphere. Four pixelwise temporal mosaicking methods, using a two years-Sentinel-2 time series and several spectral indices (NDVI, NBR2, BSI, S2WI), were developed and compared for i) pure bare soil condition (maxBSI), ii) driest soil condition (minS2WI), iii) average bare soil condition (Median) and iv) dry soil conditions excluding extreme reflectance values (R90). Three spectral modeling approaches, using the Sentinel-2 bands of the output temporal mosaics as covariates, were tested and compared: (i) Quantile Regression Forest (QRF) algorithm; (ii) QRF adding longitude and latitude as covariates (QRFxy); (iii) a hybrid approach, Linear Mixed Effect Model (LMEM), that includes spatial autocorrelation of the soil properties. We tested pairs of mosaic and spectral approaches on ten sites in Türkiye, Italy, Lithuania, and USA where soil samples were collected and SOC and clay content were measured in the lab. The average RPIQ of the best performances among the test sites was 2.50 both for SOC (RMSE = 0.15%) and clay (RMSE = 3.3%). Both accuracy level and uncertainty were mainly influenced by site characteristics of cloud frequency, soil types and management. Generally, the models including a spatial component (QRFxy and LMEM) were the best performing, while the best spatial mosaicking approaches mostly were Median and R90. The most frequent optimal combination of mosaicking and model type was Median or R90 and QRFxy for SOC, and R90 and LMEM for clay estimation.

Vision-based aerial image mosaicking algorithm with object detection

Aerial image sequence mosaicking is one of the challenging research fields in computer vision. To obtain large-scale orthophoto maps with object detection information, we propose a vision-based image mosaicking algorithm without any extra location data. According to object detection results, we define a complexity factor to describe the importance of each input Image and dynamically optimize the feature extraction process. The feature points extraction and matching processes are mainly guided by the speeded-up robust features (SURF) and the grid motion statistic (GMS) algorithm respectively. A robust reference frame selection method is proposed to eliminate the transformation distortion by searching for the center area based on overlaps. Besides, the sparse Levenberg-Marquardt (LM) algorithm and the heavy occluded frames removal method are Applied to reduce accumulated errors and further improve the mosaicking performance. The proposed algorithm is performed by using multithreading and graphics processing unit (GPU) acceleration on several aerial image datasets. Extensive experiment results demonstrate that our algorithm outperforms most of the existing aerial image mosaicking methods in visual quality while guaranteeing a high calculation speed.

Mosaicking Weather Radar Retrievals from an Operational Heterogeneous Network at C and X Band for Precipitation Monitoring in Italian Central Apennines

Meteorological radar networks are suited to remotely provide atmospheric precipitation retrieval over a wide geographic area for severe weather monitoring and near-real-time nowcasting. However, blockage due to buildings, hills, and mountains can hamper the potential of an operational weather radar system. The Abruzzo region in central Italy’s Apennines, whose hydro-geological risks are further enhanced by its complex orography, is monitored by a heterogeneous system of three microwave radars at the C and X bands with different features. This work shows a systematic intercomparison of operational radar mosaicking methods, based on bi-dimensional rainfall products and dealing with both C and X bands as well as single- and dual-polarization systems. The considered mosaicking methods can take into account spatial radar-gauge adjustment as well as different spatial combination approaches. A data set of 16 precipitation events during the years 2018–2020 in the central Apennines is collected (with a total number of 32,750 samples) to show the potentials and limitations of the considered operational mosaicking approaches, using a geospatially-interpolated dense network of regional rain gauges as a benchmark. Results show that the radar-network pattern mosaicking, based on the anisotropic radar-gauge adjustment and spatial averaging of composite data, is better than the conventional maximum-value merging approach. The overall analysis confirms that heterogeneous weather radar mosaicking can overcome the issues of single-frequency fixed radars in mountainous areas, guaranteeing a better spatial coverage and a more uniform rainfall estimation accuracy over the area of interest.

Intensity-based nonrigid endomicroscopic image mosaicking incorporating texture relevance for compensation of tissue deformation

Image mosaicking has emerged as a universal technique to broaden the field-of-view of the probe-based confocal laser endomicroscopy (pCLE) imaging system. However, due to the influence of probe-tissue contact forces and optical components on imaging quality, existing mosaicking methods remain insufficient to deal with practical challenges. In this paper, we present the texture encoded sum of conditional variance (TESCV) as a novel similarity metric, and effectively incorporate it into a sequential mosaicking scheme to simultaneously correct rigid probe shift and nonrigid tissue deformation. TESCV combines both intensity dependency and texture relevance to quantify the differences between pCLE image frames, where a discriminative binary descriptor named fully cross-detected local derivative pattern (FCLDP) is designed to extract more detailed structural textures. Furthermore, we also analytically derive the closed-form gradient of TESCV with respect to the transformation variables. Experiments on the circular dataset highlighted the advantage of the TESCV metric in improving mosaicking performance compared with the other four recently published metrics. The comparison with the other four state-of-the-art mosaicking methods on the spiral and manual datasets indicated that the proposed TESCV-based method not only worked stably at different contact forces, but was also suitable for both low- and high-resolution imaging systems. With more accurate and delicate mosaics, the proposed method holds promises to meet clinical demands for intraoperative optical biopsy.

Intensity-Mosaic: automatic panorama mosaicking of disordered images with insufficient features.

Purpose: Handling low-quality and few-feature medical images is a challenging task in automatic panorama mosaicking. Current mosaicking methods for disordered input images are based on feature point matching, whereas in this case intensity-based registration achieves better performance than feature-point registration methods. We propose a mosaicking method that enables the use of mutual information (MI) registration for mosaicking randomly ordered input images with insufficient features. Approach: Dimensionality reduction is used to map disordered input images into a low dimensional space. Based on the low dimensional representation, the image global correspondence can be recognized efficiently. For adjacent image pairs, we optimize the MI metric for registration. The panorama is then created after image blending. We demonstrate our method on relatively lower-cost handheld devices that acquire images from the retina in vivo, kidney ex vivo, and bladder phantom, all of which contain sparse features. Results: Our method is compared with three baselines: AutoStitch, "dimension reduction + SIFT," and "MI-Only." Our method compared to the first two feature-point based methods exhibits 1.25 (ex vivo microscope dataset) to two times (in vivo retina dataset) rate of mosaic completion, and MI-Only has the lowest complete rate among three datasets. When comparing the subsequent complete mosaics, our target registration errors can be 2.2 and 3.8 times reduced when using the microscopy and bladder phantom datasets. Conclusions: Using dimensional reduction increases the success rate of detecting adjacent images, which makes MI-based registration feasible and narrows the search range of MI optimization. To the best of our knowledge, this is the first mosaicking method that allows automatic stitching of disordered images with intensity-based alignment, which provides more robust and accurate results when there are insufficient features for classic mosaicking methods.

Real-Time Nonrigid Mosaicking of Laparoscopy Images.

The ability to extend the field of view of laparoscopy images can help the surgeons to obtain a better understanding of the anatomical context. However, due to tissue deformation, complex camera motion and significant three-dimensional (3D) anatomical surface, image pixels may have non-rigid deformation and traditional mosaicking methods cannot work robustly for laparoscopy images in real-time. To solve this problem, a novel two-dimensional (2D) non-rigid simultaneous localization and mapping (SLAM) system is proposed in this paper, which is able to compensate for the deformation of pixels and perform image mosaicking in real-time. The key algorithm of this 2D non-rigid SLAM system is the expectation maximization and dual quaternion (EMDQ) algorithm, which can generate smooth and dense deformation field from sparse and noisy image feature matches in real-time. An uncertainty-based loop closing method has been proposed to reduce the accumulative errors. To achieve real-time performance, both CPU and GPU parallel computation technologies are used for dense mosaicking of all pixels. Experimental results on in vivo and synthetic data demonstrate the feasibility and accuracy of our non-rigid mosaicking method.

Retinal image mosaicking using scale-invariant feature transformation feature descriptors and Voronoi diagram.

Purpose: Peripheral retinal lesions substantially increase the risk of diabetic retinopathy and retinopathy of prematurity. The peripheral changes can be visualized in wide field imaging, which is obtained by combining multiple images with an overlapping field of view using mosaicking methods. However, a robust and accurate registration of mosaicking techniques for normal angle fundus cameras is still a challenge due to the random selection of matching points and execution time. We propose a method of retinal image mosaicking based on scale-invariant feature transformation (SIFT) feature descriptor and Voronoi diagram. Approach: In our method, the SIFT algorithm is used to describe local features in the input images. Then the input images are subdivided into regions based on the Voronoi method. Each pair of Voronoi regions is matched by the method zero mean normalized cross correlation. After matching, the retinal images are mapped into the same coordinate system to form a mosaic image. The success rate and the mean registration error (RE) of our method were compared with those of other state-of-the-art methods for the P category of the fundus image registration database. Results: Experimental results show that the proposed method accurately registered 42% of retinal image pairs with a mean RE of 3.040pixels, while a lower success rate was observed in the other four state-of-the-art retinal image registration methods GDB-ICP (33%), Harris-PIIFD (0%), HM-2016 (0%), and HM-2017 (2%). Conclusions: The proposed method outperforms state-of-the-art methods in terms of quality and running time and reduces the computational complexity.

Real-time mosaicking for infrared videos from an oblique sweeping camera

Image mosaicking is widely used in Geographic Information Systems (GISs) for large-scale ground surface analysis. However, most existing mosaicking methods can only be used in offline processing due to the enormous amounts of computation. In this paper, we propose a novel and practical algorithm for real-time infrared video mosaicking. To achieve this, a novel fast template matching algorithm based on Sum of Cosine Differences (SCD) is proposed to coarsely match the sequential images. The high speed of the proposed template matching algorithm is obtained by computing correlation with Fast Fourier Transform (FFT). We also propose a novel fast Least Squares Matching (LSM) algorithm for inter-frame fine registration, which can significantly reduce the computation without degrading the matching accuracy. In addition, the proposed fast LSM can effectively adapt for noise degradation and geometric distortion. Based on the proposed fast template matching algorithm and fine registration algorithm, we develop a practical real-time mosaicking approach which can produce seamless mosaic image highly efficiently. Experiments on synthetic and real-world datasets demonstrate that the proposed algorithm is not just computationally efficient but also robust against various noise distortions.

Real-time video mosaicking to guide handheld in vivo microscopy.

Handheld and endoscopic optical-sectioning microscopes are being developed for noninvasive screening and intraoperative consultation. Imaging a large extent of tissue is often desired, but miniature in vivo microscopes tend to suffer from limited fields of view. To extend the imaging field during clinical use, we have developed a real-time video mosaicking method, which allows users to efficiently survey larger areas of tissue. Here, we modified a previous post-processing mosaicking method so that real-time mosaicking is possible at >30 frames/second when using a device that outputs images that are 400 × 400 pixels in size. Unlike other real-time mosaicking methods, our strategy can accommodate image rotations and deformations that often occur during clinical use of a handheld microscope. We perform a feasibility study to demonstrate that the use of real-time mosaicking is necessary to enable efficient sampling of a desired imaging field when using a handheld dual-axis confocal microscope.

Successive minutia‐free mosaicking for small‐sized fingerprint recognition

Small-sized fingerprint sensors, due to the convenience of integration, are widely used in many applications, especially on smart phones. However, the friction ridge information decreases with the reduction of the collected fingerprint area, resulting in degraded recognition performance. Mosaicking fingerprint impressions has been proved to be effective in boosting the recognition accuracy. Nonetheless, the minutiae-based mosaicking methods do not work well when there is no sufficient number of minutiae in the overlapping area while existing minutia-free mosaicking methods are not robust to distortion and result in low mosaicking accuracy. In this study, a novel minutia-free mosaicking algorithm used the coarse-to-fine approach is proposed to obtain a larger fingerprint impression from a couple of small-sized fingerprint impressions. It consists of three stages: an orientation field-based coarse alignment, a ridge matching-based fine alignment, and a nonlinear deformation correction with block-correspondence Thin Plate Spline model. Experimental results on the XDfinger database demonstrate that the proposed method outperforms the other six mosaicking methods in terms of reject-to-fuse rate, registration accuracy, and verification performance. Specifically, in the verification scenario, the equal error rate is reduced from 1.98% of a single impression to 0.41% of two impressions mosaicked by the authors' method.

CONSIDERING TIME IN ORTHOPHOTOGRAPHY PRODUCTION: FROM A GENERAL WORKFLOW TO A SHORTENED WORKFLOW FOR A FASTER DISASTER RESPONSE

Abstract. This article overall deals with production time with orthophoto imagery with medium size digital frame camera. The workflow examination follows two main parts: data acquisition and post-processing. The objectives of the research are fourfold: 1/ gathering time references for the most important steps of orthophoto production (it turned out that literature is missing on this topic); these figures are used later for total production time estimation; 2/ identifying levers for reducing orthophoto production time; 3/ building a simplified production workflow for emergency response: less exigent with accuracy and faster; and compare it to a classical workflow; 4/ providing methodical elements for the estimation of production time with a custom project. In the data acquisition part a comprehensive review lists and describes all the factors that may affect the acquisition efficiency. Using a simulation with different variables (average line length, time of the turns, flight speed) their effect on acquisition efficiency is quantitatively examined. Regarding post-processing, the time references figures were collected from the processing of a 1000 frames case study with 15 cm GSD covering a rectangular area of 447 km2; the time required to achieve each step during the production is written down. When several technical options are possible, each one is tested and time documented so as all alternatives are available. Based on a technical choice with the workflow and using the compiled time reference of the elementary steps, a total time is calculated for the post-processing of the 1000 frames. Two scenarios are compared as regards to time and accuracy. The first one follows the “normal” practices, comprising triangulation, orthorectification and advanced mosaicking methods (feature detection, seam line editing and seam applicator); the second is simplified and make compromise over positional accuracy (using direct geo-referencing) and seamlines preparation in order to achieve orthophoto production faster. The shortened workflow reduces the production time by more than three whereas the positional error increases from 1 GSD to 1.5 GSD. The examination of time allocation through the production process shows that it is worth sparing time in the post-processing phase.

Resource aware and incremental mosaics of wide areas from small-scale UAVs

Small-scale unmanned aerial vehicles (UAVs) are an emerging research area and have been recently demonstrated in many applications including disaster response management, construction site monitoring and wide area surveillance where multiple UAVs impose various benefits. In this work we present a system composed of multiple networked UAVs for autonomously monitoring a wide area scenario. Each UAV is able to follow waypoints and capture high-resolution images. In order to overcome the strong resource limitations we implement an incremental approach for generating an orthographic mosaic from the individual images. Captured images are pre-processed on-board, annotated with other sensor data and transferred by a prioritized transmission scheme. The ultimate goal of our approach is to generate an overview mosaic as fast as possible and to improve its quality over time. The mosaicking exploits position and orientation data of the UAV to compute rough image projections which are incrementally refined by scene structure analysis when more image data is available. We evaluate our incremental mosaicking in the strongly resource limited UAV network composed of up to three concurrently flying UAVs. Our results are compared to state-of-the-art mosaicking methods and show a unique performance in our dedicated application scenarios.

Automatic Generation of Seamline Network Using Area Voronoi Diagrams With Overlap

The mosaicking of orthoimages has been used to cover a large geographic region for various applications ranging from environmental monitoring to disaster management. However, existing mosaicking methods mainly focus on the generation of seamlines between two adjacent orthoimages. In this paper, we present a novel approach based on the use of a seamline network formed by a novel area Voronoi diagrams with overlap and the use of effective mosaic polygons (EMPs) to define the pixels of each orthoimage for the final mosaic. The generated seamline network is global based and is also optimized after refinement. It gives an effective partitioning for the regions of all orthoimages to form EMPs. The partitioning is unique, seamless, and has no redundancy. The algorithm is parallel, and the EMP of each orthoimage only has relation to orthoimages which have overlaps with it. It can ensure the flexibility and efficiency of mosaicking, without an intermediate process and independent of the sequence of the image composite. The experimental results obtained from the mosaicking of 40 color orthoimages demonstrate considerable potential for generating a seamline network automatically and effectively. This is extremely useful when a seamless mosaic is required to cover a large geographic region.

Image mosaicking for polyhedral scene and in particular singly visible surfaces

Image mosaic construction is about stitching together a number of images about the same scene to construct a single image with a larger field of view. The majority of the previous work was rooted at the use of a single image-to-image mapping termed planar homography for representing the imaged scene. However, the mapping is applicable only to cases where the imaged scene is either a single planar surface, or very distant from the cameras, or imaged under a pure rotation of the camera, and that greatly limits the range of applications of the mosaicking methods. This paper presents a novel mosaicking solution for scenes that are polyhedral (thus consisting of multiple surfaces) and that are pictured possibly in closed range of the camera. The solution has two major advantages. First, it requires only a few correspondences over the entire scene, not correspondences over every surface patch in it to work. Second, it conquers a seemingly impossible task—warping image data of surfaces that are visible in only one of the input images, which we refer to as the singly visible surfaces, to another viewpoint to constitute the mosaic there. We also provide a detail analysis of what determines whether a singly visible surface could be mosaicked or not. Experimental results on real image data are presented to illustrate the performance of the method.

Mosaicking images with parallax

In recent years, image mosaicking has been a focus of attention of many researchers and practitioners. Existing mosaicking methods are effective only in limited cases where the camera motion is almost a pure rotation or the viewed scene is planar or very far away. In this paper, we introduce a new methodology of image mosaicking for generic 3D scenes under general camera motions. What puts the mosaicking problem to a class different from those in existing work is the parallax in the images. In this paper, we develop two heuristics for performing the geometrical image transfer. The first one uses a planar motion plus parallax together with a projective reconstruction. The second one uses projective planar patches. We provide a solution for the occlusion problem introduced by the parallax field. Examples of constructing successful mosaics from real images are shown. These experiments demonstrate the feasibility of the developed methods for registering images with parallax.