Orthoimage Mosaicking Research Articles

A flexible multi-temporal orthoimage mosaicking method based on dynamic variable patches

Orthoimage mosaicking plays an important role in remote sensing applications, such as environment monitoring and protection. However, the cloud coverage in remote sensing images often results in substantial loss of information and degradation of data quality, bringing lots of challenges to orthoimage mosaicking. Therefore, this paper proposes a flexible multi-temporal orthoimage mosaicking method based on dynamic variable patches. Different from most existing pixel-based mosaicking methods, the proposed method defines patch as the minimal processing unit for orthoimage mosaicking, and the size and position of each patch are dynamic variable. Patches from different images are rearranged under the image texture consistency constraint to generate the mosaic image. In the proposed method, the initial positions of available patches of all images are determined first, then the initial size of each patch is determined by the cloud coverage and the effective area of image. Subsequently, considering the inevitable misalignments between different orthoimages, the texture similarity between patches is utilized to select the most suitable patch, and the size of the patch is also dynamically adjusted to make the misalignments within each patch controllable. Finally, considering the color inconsistency in the overlapping areas of neighboring images, an area-weighted image blending algorithm is also presented to achieve invisible seams and smooth transitions in the final mosaic. Not only does the proposed method produce mosaic images with minimal cloud coverage areas by excluding cloud pixels as much as possible, but it is also flexible for the mosaicking of images with varying degrees of misalignments, which is achieved by dynamically adjusting the size and position of each patch while maintaining texture consistency constraint. Both simulated experiments and real data experiments are carried out to verify the performance of the proposed method. The experimental results demonstrate that the proposed method can generate the mosaic image with higher spatial continuity and radiometric consistency.

Optimizing Local Alignment along the Seamline for Parallax-Tolerant Orthoimage Mosaicking

Orthoimage mosaicking with obvious parallax caused by geometric misalignment is a challenging problem in the field of remote sensing. Because the obvious objects are not included in the digital terrain model (DTM), large parallax exists in these objects. A common strategy is to search an optimal seamline between orthoimages, avoiding the majority of obvious objects. However, stitching artifacts may remain because (1) the seamline may still cross several obvious objects and (2) the orthoimages may not be precisely aligned in geometry when the accuracy of the DTM is low. While applying general image warping methods to orthoimages can improve the local geometric consistency of adjacent images, these methods usually significantly modify the geometric properties of orthophoto maps. To the best of our knowledge, no approach has been proposed in the field of remote sensing to solve the problem of local geometric misalignments after orthoimage mosaicking with obvious parallax. In this paper, we creatively propose a method to optimize local alignment along the seamline after seamline detection. It consists of the following main processes. First, we locate regions with geometric misalignments along the seamline based on the similarity measure. Second, for any one region, we find one-dimensional (1D) feature matches along the seamline using a semi-global matching approach. The deformation vectors are calculated for these matches. Third, these deformation vectors are robustly and smoothly propagated into the buffer region centered on the seamline by minimizing the associated energy function. Finally, we directly warp the orthoimages to eliminate the local parallax under the guidance of dense deformation vectors. The experimental results on several groups of orthoimages show that our proposed approach is capable of eliminating the local parallax existing in the seamline while preserving most geometric properties of digital orthophoto maps, and that it outperforms state-of-the-art approaches in terms of both visual quality and quantitative metrics.

An advanced orthoimage mosaicking method for urban areas based on edge thickening

ABSTRACTThis work is an improved version of the previous work. In order to solve the problem that the seamlines are too close to the edges of buildings, a thickened edge diagram are firstly generated and then used for searching seamlines with A* algorithm. The height gradient term is replaced by the root mean square value of height gradients term to generate thickened edge diagram. The thickened edges also bring some troubles that the edges of the adjacent buildings may be connected to each other. It will block the searching algorithm from finding a path through the two buildings. To tackle this problem, the height term is added to the cost function. By doing this, the searching algorithm will find a path across the area between two adjacent buildings since the area between the buildings have small height than on the building. We also defined constrained areas for searching the seamlines. The constrained areas make sure that the seamlines will not intersect to each other. Preliminary results show that the proposed method produces better results than the previous work by both visual and quantitative comparison.

Seamline network generation based on foreground segmentation for orthoimage mosaicking

For multiple orthoimages mosaicking, the detection of an optimal seamline in an overlapped region and the generation of a seamline network are two key issues for creating a seamless and pleasant large-scale digital orthophoto map. In this paper, a novel system is proposed to generate the large-scale orthophoto by mosaicking multiple orthoimages via Graph cuts. The proposed system is comprised of two parts. In the first part, to ensure that the detected seamline avoids crossing the obvious objects, a novel foreground segmentation-based approach is proposed to detect the optimal seamline for two adjacent images. The foreground objects are segmented from the overlapped region at the superpixel level followed by the pixel-level seamline optimization. In the second part, we propose a novel seamline network generation approach to produce the large-scale orthophoto by mosaicking multiple orthoimages. The pairwise and junction regions extracted from the initial network are refined using two-label and multi-label Graph cuts, respectively. The key advantage of our proposed seamline network is that junction points can be automatically and optimally found using the multi-label Graph cuts. The experimental results on two groups of orthoimages show that our proposed system can generate high-quality seamline networks with less artifacts, and that it outperforms the state-of-the-art algorithm and the commercial software based on visual comparison and statistical evaluation.

Optimal Seamline Detection for Orthoimage Mosaicking Based on DSM and Improved JPS Algorithm

Based on the digital surface model (DSM) and jump point search (JPS) algorithm, this study proposed a novel approach to detect the optimal seamline for orthoimage mosaicking. By threshold segmentation, DSM was first identified as ground regions and obstacle regions (e.g., buildings, trees, and cars). Then, the mathematical morphology method was used to make the edge of obstacles more prominent. Subsequently, the processed DSM was considered as a uniform-cost grid map, and the JPS algorithm was improved and employed to search for key jump points in the map. Meanwhile, the jump points would be evaluated according to an optimized function, finally generating a minimum cost path as the optimal seamline. Furthermore, the search strategy was modified to avoid search failure when the search map was completely blocked by obstacles in the search direction. Comparison of the proposed method and the Dijkstra’s algorithm was carried out based on two groups of image data with different characteristics. Results showed the following: (1) the proposed method could detect better seamlines near the centerlines of the overlap regions, crossing far fewer ground objects; (2) the efficiency and resource consumption were greatly improved since the improved JPS algorithm skips many image pixels without them being explicitly evaluated. In general, based on DSM, the proposed method combining threshold segmentation, mathematical morphology, and improved JPS algorithms was helpful for detecting the optimal seamline for orthoimage mosaicking.

Edge-Enhanced Optimal Seamline Detection for Orthoimage Mosaicking

In this letter, we propose a novel algorithm to detect the optimal seamline by using the edge-enhanced energy function for orthoimage mosaicking. First, we generate the energy cost map by using traditional intensity and gradient difference. Then, to ensure that the detected optimal seamlines avoid crossing the obvious objects, the energy map is enhanced in the regions of valid edge segments around the obvious objects. The edge segments are separately detected from input images, and the invalid edge segments are removed by using texture complexity index, image difference, and consistency constraint. Finally, we detect the optimal seamline from this edge-enhanced energy cost map via graph cuts. Experimental results on several groups of challenging orthoimages show that the proposed algorithm is capable of creating high-quality seamlines for orthoimage mosaicking and outperforms state-of-the-art algorithms and the commercial software based on the visual comparison and statistical evaluation.

Superpixel-based optimal seamline detection in the gradient domain via graph cuts for orthoimage mosaicking

ABSTRACTThis paper presents an optimal seamline detection method for orthoimage mosaicking. To ensure that the detected optimal seamlines avoid crossing many obvious objects, we first design a simple but effective criterion in the gradient domain in lieu of the traditionally used intensity domain to measure the visibility of the seam. Thereafter, we fuse this new criterion into the graph cuts energy minimisation framework to globally find the last optimal seamlines. Instead of finding the optimal solutions of seamlines in overlap regions via graph cuts among the entire set of pixels, we first find them among superpixels created from input images and then refine them in the pixel level, which greatly improves the efficiency of the global graph cuts energy optimisation because the number of elements in graph cuts dramatically decreases. Experimental results on orthoimages show that our proposed method is capable of finding high-quality seamlines for orthoimage mosaicking, and outperforms state-of-the-art algorithms and software.

Optimal Seamline Detection for Orthoimage Mosaicking by Combining Deep Convolutional Neural Network and Graph Cuts

When mosaicking orthoimages, especially in urban areas with various obvious ground objects like buildings, roads, cars or trees, the detection of optimal seamlines is one of the key technologies for creating seamless and pleasant image mosaics. In this paper, we propose a new approach to detect optimal seamlines for orthoimage mosaicking with the use of deep convolutional neural network (CNN) and graph cuts. Deep CNNs have been widely used in many fields of computer vision and photogrammetry in recent years, and graph cuts is one of the most widely used energy optimization frameworks. We first propose a deep CNN for land cover semantic segmentation in overlap regions between two adjacent images. Then, the energy cost of each pixel in the overlap regions is defined based on the classification probabilities of belonging to each of the specified classes. To find the optimal seamlines globally, we fuse the CNN-classified energy costs of all pixels into the graph cuts energy minimization framework. The main advantage of our proposed method is that the pixel similarity energy costs between two images are defined using the classification results of the CNN based semantic segmentation instead of using the image informations of color, gradient or texture as traditional methods do. Another advantage of our proposed method is that the semantic informations are fully used to guide the process of optimal seamline detection, which is more reasonable than only using the hand designed features defined to represent the image differences. Finally, the experimental results on several groups of challenging orthoimages show that the proposed method is capable of finding high-quality seamlines among urban and non-urban orthoimages, and outperforms the state-of-the-art algorithms and the commercial software based on the visual comparison, statistical evaluation and quantitative evaluation based on the structural similarity (SSIM) index.

Using vector building maps to aid in generating seams for low-attitude aerial orthoimage mosaicking: Advantages in avoiding the crossing of buildings

A novel seam detection approach based on vector building maps is presented for low-attitude aerial orthoimage mosaicking. The approach tracks the centerlines between vector buildings to generate the candidate seams that avoid crossing buildings existing in maps. The candidate seams are then refined by considering their surrounding pixels to minimize the visual transition between the images to be mosaicked. After the refinement of the candidate seams, the final seams further bypass most of the buildings that are not updated into vector maps. Finally, three groups of aerial imagery from different urban densities are employed to test the proposed approach. The experimental results illustrate the advantages of the proposed approach in avoiding the crossing of buildings. The computational efficiency of the proposed approach is also significantly higher than that of Dijkstra’s algorithm.

Seamline optimization based on ground objects classes for orthoimage mosaicking

ABSTRACTSeamline determination is one of the key steps in orthoimage mosaicking. To avoid discontinuity in mosaic, this letter presents a seamline optimization method based on ground objects classes. The presented method classifies ground objects in overlaps of adjacent images into three classes, that is, buildings, vegetation and other classes, using normalized difference vegetation index and morphological building index. Based on ground objects classes, three types of regions, that is, obstacle regions, preferred regions and general regions, are further formed. Then, each type of region is assigned a different weight to optimize obtained pixel costs. Finally, a path finding with limiting level of maximum cost and differential cost is carried out to search the shortest path from start point to end point based on optimized pixel costs in overlaps as the final seamline. Experimental results including comparison with other methods demonstrate the potential of the presented method.

A Multi-Resolution Blending Considering Changed Regions for Orthoimage Mosaicking

Blending processing based on seamlines in image mosaicking is a procedure designed to obtain a smooth transition between images along seamlines and make seams invisible in the final mosaic. However, for high-resolution aerial orthoimages in urban areas, factors such as projection differences, moving objects, and radiometric differences in overlapping areas may result in ghosting and artifacts or visible shifts in the final mosaic. Such a mosaic is not a true reflection of the earth’s surface and may have a negative impact on image interpretation. Therefore, this paper presents a multi-resolution blending method considering changed regions to improve mosaic image quality. The method utilizes the region change rate (RCR) to distinguish changed regions from unchanged regions in overlapping areas. The RCR of each region is computed using image segmentation and change detection methods. Then, a mask image is generated considering changed regions, and Gaussian and Laplacian pyramids are constructed. Finally, a multi-resolution reconstruction is performed to obtain the final mosaic. Experimental results from digital aerial orthoimages in urban areas are provided to verify this method for blending processing based on seamlines in mosaicking. Comparisons with other methods further demonstrate the potential of the presented method, as shown in a detailed comparison in three typical cases of the seamline passing by buildings, the seamline passing through buildings, and the seamline passing through areas with large radiometric differences.

DETECTION OF ORTHOIMAGE MOSAICKING SEAMLINES BY MEANS OF WAVELET TRANSFORM

Abstract. The detection of orthoimage mosaicking seamlines by means of wavelet transform was examined. Radiometric alignment was omitted, giving priority to the issue of seamlines which bypass locations where there is a parallax between orthoimages. The importance of this issue is particularly relevant for images with very high resolution. In order to create a barrier image between orthoimages, the redundant wavelet transform variant known as MODWT-MRA was used. While more computationally complex than the frequently used DWT, it enables very good multiresolution edge detection. An IT prototype was developed on the basis of the described concept, and several cases of seamline detection were tested on the basis of data with a resolution of 10 cm to 1 m. The correct seamline location was obtained for each test case. This result opens the door to future expansion of the radiometric alignment method, which is also based on wavelets.

DETECTION OF ORTHOIMAGE MOSAICKING SEAMLINES BY MEANS OF WAVELET TRANSFORM

The detection of orthoimage mosaicking seamlines by means of wavelet transform was examined. Radiometric alignment was omitted, giving priority to the issue of seamlines which bypass locations where there is a parallax between orthoimages. The importance of this issue is particularly relevant for images with very high resolution. In order to create a barrier image between orthoimages, the redundant wavelet transform variant known as MODWT-MRA was used. While more computationally complex than the frequently used DWT, it enables very good multiresolution edge detection. An IT prototype was developed on the basis of the described concept, and several cases of seamline detection were tested on the basis of data with a resolution of 10 cm to 1 m. The correct seamline location was obtained for each test case. This result opens the door to future expansion of the radiometric alignment method, which is also based on wavelets.

A FRAMEWORK FOR AN AUTOMATIC SEAMLINE ENGINE

Seamline generation is a crucial last step in the ortho-image mosaicking process. In particular, it is required to convolute residual geometric and radiometric imperfections that stem from various sources. In particular, temporal differences in the acquired data will cause the scene content and illumination conditions to vary. These variations can be modelled successfully. However, one is left with micro-differences that do need to be considered in seamline generation. Another cause of discrepancies originates from the rectification surface as it will not model the actual terrain and especially human-made objects perfectly. Quality of the image orientation will also contribute to the overall differences between adjacent ortho-rectified images. <br><br> Our approach takes into consideration the aforementioned differences in designing a seamline engine. We have identified the following essential behaviours of the seamline in our engine: 1) Seamlines must pass through the path of least resistance, i.e., overlap areas with low radiometric differences. 2) Seamlines must not intersect with breaklines as that will lead to visible geometric artefacts. And finally, 3), shorter seamlines are generally favourable; they also result in faster operator review and, where necessary, interactive editing cycles. The engine design also permits alteration of the above rules for special cases. <br><br> Although our preliminary experiments are geared towards line imaging systems (i.e., the Leica ADS family), our seamline engine remains sensor agnostic. Hence, our design is capable of mosaicking images from various sources with minimal effort. The main idea behind this engine is using graph cuts which, in spirit, is based of the max-flow min-cut theory. The main advantage of using graph cuts theory is that the generated solution is global in the energy minimization sense. In addition, graph cuts allows for a highly scalable design where a set of rules contribute towards a cost function which, in turn, influences the path of minimum resistance for the seamlines. In this paper, the authors present an approach for achieving quality seamlines relatively quickly and with emphasis on generating truly seamless ortho-mosaics.

A FRAMEWORK FOR AN AUTOMATIC SEAMLINE ENGINE

Abstract. Seamline generation is a crucial last step in the ortho-image mosaicking process. In particular, it is required to convolute residual geometric and radiometric imperfections that stem from various sources. In particular, temporal differences in the acquired data will cause the scene content and illumination conditions to vary. These variations can be modelled successfully. However, one is left with micro-differences that do need to be considered in seamline generation. Another cause of discrepancies originates from the rectification surface as it will not model the actual terrain and especially human-made objects perfectly. Quality of the image orientation will also contribute to the overall differences between adjacent ortho-rectified images. Our approach takes into consideration the aforementioned differences in designing a seamline engine. We have identified the following essential behaviours of the seamline in our engine: 1) Seamlines must pass through the path of least resistance, i.e., overlap areas with low radiometric differences. 2) Seamlines must not intersect with breaklines as that will lead to visible geometric artefacts. And finally, 3), shorter seamlines are generally favourable; they also result in faster operator review and, where necessary, interactive editing cycles. The engine design also permits alteration of the above rules for special cases. Although our preliminary experiments are geared towards line imaging systems (i.e., the Leica ADS family), our seamline engine remains sensor agnostic. Hence, our design is capable of mosaicking images from various sources with minimal effort. The main idea behind this engine is using graph cuts which, in spirit, is based of the max-flow min-cut theory. The main advantage of using graph cuts theory is that the generated solution is global in the energy minimization sense. In addition, graph cuts allows for a highly scalable design where a set of rules contribute towards a cost function which, in turn, influences the path of minimum resistance for the seamlines. In this paper, the authors present an approach for achieving quality seamlines relatively quickly and with emphasis on generating truly seamless ortho-mosaics.

Seamline Determination for High Resolution Orthoimage Mosaicking Using Watershed Segmentation

Image mosaicking is a process during which multiple orthoimages are combined into a single seamless composite orthoimage. One of the most difficult steps in the automatic mosaicking of orthoimages is the seamline determination. This paper presents a novel algorithm that selects seamlines based on marker-based watershed segmentation. A representative seamline is extracted at the object level and the pixel level as follows. First, a watershed segmentation is performed to obtain the objects. To avoid over-segmentation, a regional adaptive marker-based watershed segmentation is proposed. Second, the object difference estimated by the correlation coefficient of each object is calculated, and the region adjacency matrix is built. Third, a technique for minimizing the maximum object cost is adopted to determine the objects through which the seamlines pass. Finally, pixel-level optimization is performed using Dijkstra's algorithm with a binary min-heap to determine the final seamlines. The experimental results on digital aerial orthoimages in different areas demonstrate the feasibility and effectiveness of the proposed method compared with other algorithms.

Region change rate-driven seamline determination method

This paper presents a novel method of seamline determination using region change rate (RCR) for orthoimage mosaicking. RCR is the change rate of the segmented region and is calculated based on the percentage of changed pixels in the segmented region. The presented method contains a two-level seamline optimization procedure to determine the final seamline using RCR. First, an object-level optimization based on image segmentation and change detection is executed to determine the allowed regions, i.e., the connected regions with minimized maximum RCR value through which seamlines are allowed to pass. This optimization is used to limit the maximum RCR along the seamline. Second, to determine the optimized seamline, pixel-level optimization is performed using Dijkstra’s algorithm based on differential cost. Finally, experimental results from digital aerial orthoimages and comparisons with other methods demonstrate the potential of the presented method for seamline determination.

Seamline Determination Based on Segmentation for Urban Image Mosaicking

This letter presents a method of seamline determination based on segmentation for orthoimage mosaicking in an urban area. Image segmentation is used to achieve regions of objects. First, preferred regions through which seamlines are inclined to be passed are determined by spans of segmented regions. Second, pixel-level optimization is carried out using Dijkstra's algorithm based on differential cost to find the optimized seamline. The experimental results on digital aerial orthoimages in the urban area prove that the new method is promising for the seamline determination in mosaicking.

Seamline detection in colour orthoimage mosaicking by use of twin snakes

In the last step of the mosaic production chain, neighbouring and partly overlapping orthoimages of a scene are merged to one mosaic. This should be done in a way that the transition from one to another orthoimage cannot be seen. The production line of orthoimages consists of several steps, each of which can introduce a different appearance regarding geometry, radiometry and spectral properties to the resulting orthoimage. For mosaicking adjacent orthoimages, a path of lowest difference in a combination of criteria is searched in the overlap area of these images. The seamline is chosen along this path of maximum similarity. In this paper, criteria for such an optimal seamline in colour orthoimages are elaborated. The main requirements are on one hand high colour similarity of the images (mainly in hue and intensity), and on the other hand high texture similarity (in orientation and magnitude of image gradients). The specified criteria are formulated in the energy function of snakes. A snake is an active contour which moves through an image and changes its shape until a minimum of its energy function is found. We use two snakes that attract one another (twin snakes). In a hierarchical strategy, a proper seamline is delineated fully automatically. The potential of the method is shown with an example.