Stereo Images Research Articles

Efficient multi-task progressive learning for semantic segmentation and disparity estimation

Scene understanding is an important area in robotics and autonomous driving. To accomplish these tasks, the 3D structures in the scene have to be inferred to know what the objects and their locations are. To this end, semantic segmentation and disparity estimation networks are typically used, but running them individually is inefficient since they require high-performance resources. A possible solution is to learn both tasks together using a multi-task approach. Some current methods address this problem by learning semantic segmentation and monocular depth together. However, monocular depth estimation from single images is an ill-posed problem. A better solution is to estimate the disparity between two stereo images and take advantage of this additional information to improve the segmentation. This work proposes an efficient multi-task method that jointly learns disparity and semantic segmentation. Employing a Siamese backbone architecture for multi-scale feature extraction, the method integrates specialized branches for disparity estimation and coarse and refined segmentations, leveraging progressive task-specific feature sharing and attention mechanisms to enhance accuracy for solving both tasks concurrently. The proposal achieves state-of-the-art results for joint segmentation and disparity estimation on three distinct datasets: Cityscapes, TrimBot2020 Garden, and S-ROSeS, using only 1/3 of the parameters of previous approaches.

Impact of Molar Composition on the Functional Properties of Glutinous Rice Starch-Chitosan Blend: Natural-Based Active Coating for Extending Mango Shelf Life.

This study investigates natural-based blends of glutinous rice starch (GRS) and chitosan (CS), varying their molar composition (0:100, 30:70, 50:50, 70:30, and 100:0) to explore their interaction dynamics. Our findings illustrate the versatility of these blends in solution and film forms, offering applications across diverse fields. Our objective is to understand their impact on coatings designed to extend the post-harvest shelf life of mangoes. Results reveal that increasing chitosan content in GRS/CS blends enhances mechanical strength, hydrophobicity, and resistance to Colletotrichum gloeosporioides infection, a common cause of mango anthracnose. These properties overcome limitations of GRS films. Advanced techniques, including FTIR analysis and stereo imaging, confirmed robust interaction between GRS/CS blend films and mango cuticles, improving coverage with higher chitosan content. This comprehensive coverage reduces mango dehydration and respiration, thereby preserving quality and extending shelf life. Coating with a GRS/CS blend containing at least 50% chitosan effectively prevents disease progression and maintains quality over a 10-day storage period, while uncoated mangoes fail to meet quality standards within 2 days. Moreover, increasing the starch proportion in GRS/CS blends enhances film density, optical properties, and reduces reliance on acidic solvents, thereby minimizing undesirable changes in product aroma and taste.

Qinghai-Tibet Plateau Glacier-DEM product derived from the Chinese stereo mapping satellites

Abstract. The Digital Elevation Model (DEM) is an important basic data for glacier research on the Qinghai-Tibet Plateau. With the rapid development of China 's stereo mapping satellite, it is possible to obtain high-precision DEM in the glacier area of the Qinghai-Tibet Plateau. In this study, the stereo images and laser altimetry data of ZY-3 and GF-7 satellites were comprehensively used, and the matching and joint adjustment methods of laser elevation control points and stereo images were proposed. A high-precision DEM product in glacier area was established, and the elevation accuracy of DEM was verified by using UAV data and ICESat-2 data. The results show that the accuracy of GF-7 DEM is better than 1.5 m when the slope is less than 6 degrees, and the overall accuracy is better than 2.8 m. The accuracy of ZY-3 DEM is better than 3.5 m when the slope is less than 6 degrees, and the overall accuracy is better than 5.0 m. Compared with the open source DEM with medium spatial resolution, the DEM elevation accuracy based on domestic stereo mapping satellite images is better, and the grid is finer and more detailed to describe the texture characteristics of the end of the glaciers. Compared with the high spatial resolution data set HMA DEM, the ZY-3 DEM is slightly worse and the GF-7 DEM is better, and both the ZY-3 DEM and GF-7 DEM are better than the HMA DEM in terms of coverage integrity in the Qinghai- Tibet Plateau. Chinese stereo mapping satellites can generate high-precision DEM in the glacier area, and can provide more accurate and reliable terrain reference data than the existing open free data for the study of glaciers in the Qinghai-Tibet Plateau.

Research on Monitoring and Application of Ecological Restoration Engineering in Open Pit Backfilling Mines Based on Satellite Remote Sensing Data

Abstract. The restoration and management of mining ecological environment is an important component of China's ecological civilization construction. The implementation of restoration projects is to avoid the serious impact caused by the continuous deterioration of mining ecological structure, and is a key way to maintain regional ecological security and energy and mineral security. Therefore, efficient and rapid monitoring and evaluation of the effectiveness of mining ecological restoration is an essential and important link. With the rapid development of remote sensing technology, remote sensing monitoring has become an important means to objectively, quickly, and accurately obtaining changes in the mining environment. It can provide timely and long-term monitoring of mining and backfilling conditions in mining areas. This article uses the multi temporal stereo images of the Resource 3 satellite over four years to monitor the mining and backfilling situation of the Zhungeer open-pit coal mine area in Inner Mongolia. By reconstructing the Digital Surface Model (DSM) of the open-pit mining area, the changes and thresholds of the multi temporal DSM data are statistically analyzed to extract the mining and backfilling areas, and the earthwork volume is calculated. The results showed that from 2013 to 2016, the main large-scale mining faces in the study area of Zhungeer open-pit mining area had a total mining operation of about 563.5 million cubic meters, and a total backfilling operation of about 604.29 million cubic meters; During the period from 2016 to 2018, the main large-scale mining faces in the region had a total of approximately 721.71 million cubic meters of mining operations and a total of approximately 805.42 million cubic meters of backfilling operations. The overall operational intensity increased from 2016 to 2018. The research results show that based on satellite image data, it is convenient and efficient to obtain DSM and corresponding changes of multiple time periods in mining areas. Monitoring results can provide important support for regional environmental governance and protection, as well as safety production in mining areas.

Study on the Application of Water Resource Management Based on GF-7 Stereo Mapping Satellite

Abstract. GF-7 satellite was successfully launched on 3 Nov 2019. The two-line stereo camera can effectively obtain 20 km-width panchromatic stereo images with resolution better than 0.8m and 3.2m-resolution multispectral images. Through the composite mapping mode of stereo camera and laser altimeter, the satellite realizes 1:10,000-scale stereo mapping. This article selected the Qiafuqihai Reservoir and its upstream of the Yili River Basin as the study area. Hydrological landforms and vegetation coverage were monitored and three-dimensional dynamic simulation software was developed to verify the potential application of GF-7 stereo mapping satellite data in supporting drainage basin water resource allocation and management in future scenarios. 3 meters DSM derived from GF-7 surveying and mapping satellite finely portrayed the characteristics of hydroponic landforms of small watersheds. The upstream of Qiafuqihai was divided into eleven small watersheds. The water density was 8.5 times more than which was produced from STRM 90m. The elevation of the water-level fluctuation zone of the Qiafuqihai Reservoir ranged from 970 to 998 meters with the water area changed from 28.3 to 57.6 square kilometres. The terrain in the northwest and northeast was flat, with the main vegetation types being natural grasslands (64.9%) and arid lands (5.03%). The three-dimensional dynamic visualization software was developed and demonstrated hydrology information, vegetation coverage information and dynamic changes in the landform of the basin in three-dimensional environment. GF-7, the stereo surveying and mapping satellite, could be perfectly able to support the application of water resources deployment and management in the future.

Building Height Extraction Based on Satellite GF-7 High-Resolution Stereo Image

Abstract. High-resolution remote sensing images can distinguish objects of smaller size, so as to more clearly express the texture features and structural information of objects, and provide a data source for the development of large-scale mapping, high-precision stereo measurement and other fields. The purpose of this paper is to complete the height estimation of the buildings by analyzing the stereoscopic observation formed by the front and rear view images of the Gaofen-7 line array CCD. After determining the roof profile of the building on the rear-view image, assuming a series of object elevations of the building, that is, searching for elevations with a certain step distance within a certain elevation search range, adopt the object-based image matching VLL algorithm, Through the RFM imaging model of the Gaofen-7 sensor, the rear-view contour is projected to the front-view image, and then the PSNR is selected as the similarity measure of the window, and the similarity of the image block area corresponding to the front- and rear-view contour is calculated. Corresponds to the hypothetical object elevation as the estimated height of the building. Under the technical route of this paper, the height of buildings on high-resolution images can be estimated to a level within 3 meters of accuracy.

CVGSR: Stereo image Super-Resolution with Cross-View guidance

The ability to capture the complementary information of stereo images is critical to the development of stereo image super-resolution. Most existing studies have attempted to integrate reliable stereo correspondence along the polar directions through parallax attention and various fusion strategies. However, most of these approaches ignore the large parallax differences in stereo images, resulting in poor performance of convolutional-based parallax attention in capturing the long-range dependencies between images. In this paper, we propose a novel cross-view guided stereo image super-resolution network (CVGSR) for reconstructing high-resolution stereo image pairs with rich texture details by fully exploiting the complementary nature of stereo image pairs. Specifically, we first deploy a cross-view interaction module (CVIM) to explore intra/cross-view dependencies from local to global to compensate for the incomplete compatibility of information between the left and right views. This module uses a progressive cross-guiding strategy to better merge features from occluded and non-occluded regions. Based on this, an efficient attention Transformer (EAT) is improved to activate more input information and further mine the cross-view complementarity. Furthermore, we design a texture loss to optimize the visual perceptual quality of reconstructed images with sharp boundaries and rich texture details. Extensive experiments on four stereo image datasets demonstrate that the proposed CVGSR achieves a competitive and excellent performance.

Application of image rendering based on improved neural networks and sensors in landscape design

Landscape planning and design is an indispensable part of modern urban construction. In landscape design, rendering technology can transform designers' imagination into actual images, allowing people to better understand the design intent. In order to solve the problem of insufficient precision in image rendering in landscape design, this paper proposes a landscape design image rendering technology based on improved neural network technology. This technology combines stereo matching algorithms, which evaluate the corrected stereo images to calculate the differences between the real scene and the underlying model. The stereo matching algorithm is based on the basic model and obtains accurate depth information from a wide range image pair, making the rendered landscape image more realistic and precise. The experiment demonstrates the effectiveness of the landscape design image rendering system through two important aspects: processing speed and parallax accuracy. The development of image rendering technology in landscape design cannot be separated from the mutual promotion of theory and practice. The technology proposed in this article has certain practical value and hopes to make more positive contributions to urban construction and cultural inheritance.

A road surface reconstruction dataset for autonomous driving

Recent developments in intelligent robot systems, especially autonomous vehicles, put forward higher requirements for safety and comfort. Road conditions are crucial factors affecting the comprehensive performance of ground vehicles. Nonetheless, existing environment perception datasets for autonomous driving lack attention to road surface areas. In this paper, we introduce the road surface reconstruction dataset, providing multi-modal, high-resolution, and high-precision data collected by real-vehicle platform in diverse driving conditions. It covers common road types containing approximately 16,000 pairs of stereo images, point clouds, and ground-truth depth/disparity maps, with accurate data processing pipelines to ensure its quality. Preliminary evaluations reveal the effectiveness of our dataset and the challenge of the task, underscoring substantial opportunities of it as a valuable resource for advancing computer vision techniques. The reconstructed road structure and texture contribute to the analysis and prediction of vehicle responses for motion planning and control systems.

Ultrabright Dibenzofluoran-Based Polymer Dots with NIR-IIa Emission Maxima and Unusual Large Stokes Shifts for 3D Rotational Stereo Imaging.

Emerging organic molecules with emissions in the second near-infrared (NIR-II) region are garnering significant attention. Unfortunately, achieving accountable organic emission intensity over the NIR-IIa (1300nm) region faces challenges due to the intrinsic energy gap law. Up to the current stage, all reported organic NIR-IIa emitters belong to polymethine-based dyes with small Stokes shifts (<50nm) and low quantum yield (QY; ≤0.015%). However, such polymethines have proved to cause self-absorption with constrained emission brightness, limiting advanced development in deep-tissue imaging. Here a new NIR-IIa scaffold based on rigid and highly conjugated dibenzofluoran core terminated by amino-containing moieties that reveal emission peaks of 1230-1305nm is designed. The QY is at least 10 times higher than all synthesized or reported NIR-IIa polymethines with extraordinarily large Stokes shifts of 370-446nm. DBF-BJ is further prepared as a polymer dot to demonstrate its in vivo 3D stereo imaging of mouse vasculature with a 1400nm long-pass filter.

Spectral or Dynamic Constellations as Sound Sources in Maps

Research on musical aesthetic patterns on different decades of the second part of the last twentieth century allowed specific studies based on frequency and amplitude data. This approach is carried out mainly to locate musical instruments and their relations inside the stereo image (the traditional audio virtual space), so a bidimensional map development was determined as a key part for finding spatial distribution patterns on different periods of stereo music masters -to be able to characterize timbral aesthetics from a measured perspective-. This article presents the concepts, methodology and the analysis performed, a well all their results. Hence, a new approach of musical sound sources conceptualization inside the stereo image is presented, where they are either spectral or dynamic component constellations. Additionally, this concept and approach could apply to any kind of sound sources inside a recorded signa

Computational 3D topographic microscopy from terabytes of data per sample

We present a large-scale computational 3D topographic microscope that enables 6-gigapixel profilometric 3D imaging at micron-scale resolution across >110 cm2 areas over multi-millimeter axial ranges. Our computational microscope, termed STARCAM (Scanning Topographic All-in-focus Reconstruction with a Computational Array Microscope), features a parallelized, 54-camera architecture with 3-axis translation to capture, for each sample of interest, a multi-dimensional, 2.1-terabyte (TB) dataset, consisting of a total of 224,640 9.4-megapixel images. We developed a self-supervised neural network-based algorithm for 3D reconstruction and stitching that jointly estimates an all-in-focus photometric composite and 3D height map across the entire field of view, using multi-view stereo information and image sharpness as a focal metric. The memory-efficient, compressed differentiable representation offered by the neural network effectively enables joint participation of the entire multi-TB dataset during the reconstruction process. Validation experiments on gauge blocks demonstrate a profilometric precision and accuracy of 10 µm or better. To demonstrate the broad utility of our new computational microscope, we applied STARCAM to a variety of decimeter-scale objects, with applications ranging from cultural heritage to industrial inspection.

Stereo matching from monocular images using feature consistency

AbstractSynthetic images facilitate stereo matching. However, synthetic images may suffer from image distortion, domain bias, and stereo mismatch, which would significantly restrict the widespread use of stereo matching models in the real world. The first goal in this paper is to synthesize real‐looking images for minimizing the domain bias between the synthesized and real images. For this purpose, sharpened disparity maps are produced from a mono real image. Then, stereo image pairs are synthesized using these imperfect disparity maps and the single real image in the proposed pipeline. Although the synthesized images are as realistic as possible, the domain styles of the synthesized images are always very different from the real images. Thus, the second goal is to enhance the domain generalization ability of the stereo matching network. For that, the feature extraction layer is replaced with a teacher–student model. Then, a constraint of binocular contrast features is imposed on the output of the model. When tested on the KITTI, ETH3D, and Middlebury datasets, the accuracy of the method outperforms traditional methods by at least 30%. Experiments demonstrate that the approaches are general and can be conveniently embedded into existing stereo networks.

Urban Building Height Extraction from Gaofen-7 Stereo Satellite Images Enhanced by Contour Matching

The traditional method for extracting the heights of urban buildings involves utilizing dense matching algorithms on stereo images to generate a digital surface model (DSM). However, for urban buildings, the disparity discontinuity issue that troubles the dense matching algorithm makes the elevations of high-rise buildings and the surrounding areas inaccurate. The occlusion caused by trees in greenbelts makes it difficult to accurately extract the ground elevation around the building. To tackle these problems, a method for building height extraction from Gaofen-7 (GF-7) stereo images enhanced by contour matching is presented. Firstly, a contour matching algorithm was proposed to extract accurate building roof elevation from GF-7 images. Secondly, a ground filtering algorithm was employed on the DSM to generate a digital elevation model (DEM), and ground elevation can be extracted from this DEM. The difference between the rooftop elevation and the ground elevation represents the building height. The presented method was verified in Yingde, Guangzhou, Guangdong Province, and Xi’an, Shaanxi Province. The experimental results demonstrate that our proposed method outperforms existing methods in building height extraction concerning accuracy.

A dense matching method for remote sensing images fused with CPS denoising

Dense matching of remote sensing images is crucial for 3D reconstruction. This study proposes an enhanced dense matching method employing the CPS image denoising algorithm, aiming to boost the SGM algorithm's accuracy and efficiency in remote sensing image matching. The stereo image pair's quality is evaluated using the PSNR index, and a decision-making criterion based on the CPS algorithm is incorporated to determine the need for denoising. Preprocessing steps, including image cropping and pixel coordinate transformation, significantly reduce computational requirements. An epipolar line model, minimizing the disparity between two pixels, is used for calculations. This model is employed to construct an epipolar image, enhancing the accuracy and efficiency of the process. The study conducted experimental validation and analysis of the mismatch rate, running time, and denoising effect of the algorithm using the Middlebury 2021 stereo datasets. Additionally, the matching results of the World-View3 satellite stereo image pairs were visualized and analyzed. The experimental results indicate that the proposed algorithm reduces the average mismatch rate by 13.1% and increases the running speed by about 3 to 4 times compared to the SGBM algorithm. Specifically, the denoising effect reduces the mismatch rate of the reconstructed image by an average of 8.97%. The results indicate that the CPS method effectively addresses dense matching challenges in the presence of image blur and noise, thereby improving the operational efficiency and accuracy of the dense matching algorithm.

Geo-positioning Method of Optical Satellite Unconventional Stereo Imagery Based on Optimal Intersection

Generally, the classic three-dimensional (3D) geometric positioning of optical satellite imagery uses the least squares principle to calculate the coordinates of a ground point by minimizing the sum of the squares of the distances between two imaging rays, which requires the standard stereo data with good imaging geometric conditions. As for unconventional stereo images, the undesirable and ubiquitous weak intersection phenomena exist in data will lead to bad results or even calculation failures for the conventional method. By selecting the highest precision intersection point in block adjustment, a new method that can solve the 3D coordinates with higher accuracy and stability was proposed. Tests of two data sets covering different landscapes validated the effectiveness of the method. The results showed that the geo-positioning performance and robustness of the proposed method was better than that of the conventional method, and this advantage is even greater in areas with more undulating terrain and more images with weak convergence.

An efficient and accurate multi-level cascaded recurrent network for stereo matching

With the advent of Transformer-based convolutional neural networks, stereo matching algorithms have achieved state-of-the-art accuracy in disparity estimation. Nevertheless, this method requires much model inference time, which is the main reason limiting its application in many vision tasks and robots. Facing the trade-off problem between accuracy and efficiency, this paper proposes an efficient and accurate multi-level cascaded recurrent network, LMCR-Stereo. To recover the detailed information of stereo images more accurately, we first design a multi-level network to update the difference values in a coarse-to-fine recurrent iterative manner. Then, we propose a new pair of slow-fast multi-stage superposition inference structures to accommodate the differences between different scene data. Besides, to ensure better disparity estimation accuracy with faster model inference speed, we introduce a pair of adaptive and lightweight group correlation layers to reduce the impact of erroneous rectification and significantly improve model inference speed. The experimental results show that the proposed approach achieves a competitive disparity estimation accuracy with a faster model inference speed than the current state-of-the-art methods. Notably, the model inference speed of the proposed approach is improved by 46.0% and 50.4% in the SceneFlow test set and Middlebury benchmark, respectively.

Stereo Imaging with a Hemispherical Field-of-View Metalens Camera

Stereo Imaging with a Hemispherical Field-of-View Metalens Camera

Irregular boundaries stereo images dataset creating using depth estimation model

AbstractThis paper introduces a stereoscopic image and depth dataset created using a deep learning model. It addresses the challenge of obtaining accurate and annotated stereo image pairs with irregular boundaries for deep learning model training. Stereoscopic image and depth dataset provides a unique resource for training deep learning models to handle irregular boundary stereoscopic images, which are valuable for real-world scenarios with complex shapes or occlusions. The dataset is created using monocular depth estimation, a state-of-the-art depth estimation model, and it can be used in applications like rectifying images, estimating depth, detecting objects, and autonomous driving. Overall, this paper presents a novel dataset that demonstrates its effectiveness and potential for advancing stereo vision and developing deep learning models for computer vision applications.

Bridging local and global representations for self-supervised monocular depth estimation

Monocular depth estimation is a challenging problem, especially in a self-supervised manner without relying on the depth ground truth. The self-supervised approach places higher demands on the global and local feature extraction capabilities of the network. Based on this view, we propose to perform efficient hybrid feature extraction by exploiting the capacity of the Transformer and convolutional neural network to model long-range dependencies and local correlations at the same time. A bowknot-type fuser is designed to align features extracted from different sources as well as bridge global and local semantic representations. To obtain higher-quality pseudo-labels from the extracted features, we suggest the pseudo-label smoothing technique to fully utilize the multi-scale features, consequently boosting the effect of self-distillation loss as an auxiliary supervision to train the neural network. In addition, we propose pixel adaptive smoothness loss to refine the predicted depth map by introducing the image’s textural and spatial information. The suggested method is trained on the KITTI benchmark using stereo image pairs and achieves competitive depth estimation performance in contrast with previous approaches. The code and models are available at https://github.com/MaylingLin/BLGR-Depth.