Scene Images Research Articles

Automatic extraction and 3D modeling of real road scenes using UAV imagery and deep learning semantic segmentation

ABSTRACT The extraction of roads from UAV images is challenged by lighting, noise, occlusions, and similar non-road objects, making high-quality road extraction difficult. To addressing these issues, this study proposes an enhanced U-Net network to automate the extraction and 3D modeling of real road scenes using UAV imagery. Initially, a cascaded atrous spatial pyramid module was integrated into the encoder to capitalize on global context information, thereby refining the fuzzy segmentation outcomes. Subsequently, a module for augmenting road feature extraction was added within the channel, and a spatial attention mechanism was introduced in the decoder to enhance edge clarity. Experimental results demonstrated that this model captures more road information compared to mainstream networks and effectively incorporates topological structure perception for road extraction in complex scenarios, thus improving road connectivity. The model achieved an F1 score and mean Intersection over Union (mIoU) of 85.6% and 81.2%, respectively, on UAV images of road scenes – marking improvements of 3.9% and 3.4% over the traditional U-Net model, thereby exhibiting superior automatic road extraction capabilities. Ultimately, the model facilitated refined modeling and visual analysis of road scenes, achieving high overall accuracy and detailed local restoration of the actual scene.

A new snow cover mapping algorithm for Chinese geostationary meteorological satellite FY-4A AGRI data

ABSTRACT China’s second generation of geostationary meteorological satellites Fengyun-4 (FY-4) are equipped with the Advanced Geosynchronous Radiation Imager (AGRI) which has the advantages of large number of spectral bands and high temporal resolution. To fully leverage these features, a new snow cover (SC) mapping algorithm was proposed, which firstly derives SC map from a single scene image and then combines multi-temporal SC results within one day into a daily SC composite. The novelty of the proposed algorithm includes the utilization of small spatial and temporal divisions to avoid the use of complex kernel-driven models for the correction of bidirectional effects of surface reflectance and a novel framework to merge multi-temporal SC maps by weighting the solar illumination conditions. The SC results were validated against in-situ snow depth measurements, Landsat 8 derived SC data and MODIS SC products. Results indicate that merging multi-temporal SC data can reduce cloud obscuration with the average daily cloud coverage percentage decreasing from about 47.03% for single scene to 35.62% for merged one and slightly improve snow detection accuracy. The overall accuracy (OA) ranges from 95.00% to 96.19%, and the F-score (FS) varies from 77.78% to 87.14% when compared to different reference data, suggesting an overall good performance.

Predict Future Transient Fire Heat Release Rates Based on Fire Imagery and Deep Learning

The fire heat release rate (HRR) is a crucial parameter for describing the combustion process and its thermal effects. In recent years, some studies have employed fire scene images and deep learning algorithms to predict real-time fire HRR, which has led to the advancement of HRR prediction in terms of both lightweightness and real-time monitoring. Nevertheless, the development of an early-stage monitoring system for fires and the ability to predict future HRR based on current moment data represents a crucial foundation for evaluating the scale of indoor fires and enhancing the capacity to prevent and control such incidents. This paper proposes a deep learning model based on continuous fire scene images (containing both flame and smoke features) and their time-series information to predict the future transient fire HRR. The model (Att-BiLSTM) comprises three bi-directional long- and short-term memory (Bi-LSTM) layers and one attention layer. The model employs a bidirectional feature extraction approach, followed by the introduction of an attention mechanism to highlight the image features that have a critical impact on the prediction results. In this paper, a large-scale dataset is constructed by collecting 27,231 fire scene images with instantaneous HRR annotations from 40 different fire trials from the NIST database. The experimental results demonstrate that Att-BiLSTM is capable of effectively utilizing fire scene image features and temporal information to accurately predict future transient HRR, including those in high-brightness fire environments and complex fire source situations. The research presented in this paper offers novel insights and methodologies for fire monitoring and emergency response.

Non-Line-of-Sight Imaging and Vibrometry Using a Comb-Calibrated Coherent Sensor.

Non-line-of-sight (NLOS) imaging has the ability to reconstruct hidden objects, allowing a wide range of applications. Existing NLOS systems rely on pulsed lasers and time-resolved single-photon detectors to capture the information encoded in the time of flight of scattered photons. Despite remarkable advances, the pulsed time-of-flight LIDAR approach has limited temporal resolution and struggles to detect the frequency-associated information directly. Here, we propose and demonstrate the coherent scheme-frequency-modulated continuous wave calibrated by optical frequency comb-for high-resolution NLOS imaging, velocimetry, and vibrometry. Our comb-calibrated coherent sensor presents a system temporal resolution at subpicosecond and its superior signal-to-noise ratio permits NLOS imaging of complex scenes under strong ambient light. We show the capability of NLOS localization and 3D imaging at submillimeter scale and demonstrate NLOS vibrometry sensing at an accuracy of dozen Hertz. Our approach unlocks the coherent LIDAR techniques for widespread use in imaging science and optical sensing.

Efficient multi-scale network for semantic segmentation of fine-resolution remotely sensed images

Abstract Semantic segmentation of remote sensing urban scene images has diverse practical applications, including land cover mapping, urban change detection, environmental protection, and economic evaluation. However, classical semantic segmentation networks encounter challenges such as inadequate utilization of multi-scale semantic information and imprecise edge target segmentation in high-resolution remote sensing images. In response, this article introduces an efficient multi-scale network (EMNet) tailored for semantic segmentation of common features in remote sensing images. To address these challenges, EMNet integrates several key components. Firstly, the efficient atrous spatial pyramid pooling module is employed to enhance the relevance of multi-scale targets, facilitating improved extraction and processing of context information across different scales. Secondly, the efficient multi-scale attention mechanism and multi-scale jump connections are utilized to fuse semantic features from various levels, thereby achieving precise segmentation boundaries and accurate position information. Finally, an encoder-decoder structure is incorporated to refine the segmentation results. The effectiveness of the proposed network is validated through experiments conducted on the publicly available DroneDeploy image dataset and Potsdam dataset. Results indicate that EMNet achieves impressive performance metrics, with mean intersection over union (MIoU), mean precision (MPrecision), and mean recall (MRecall) reaching 75.99%, 86.76%, and 85.07%, respectively. Comparative analysis demonstrates that the network proposed in this article outperforms current mainstream semantic segmentation networks on both the DroneDeploy and Potsdam dataset.

Attractive appearance, future wealth! How exposure to visual wealth on social media contributes to female self-objectification

Exposure to visual wealth on social media is a common phenomenon in daily life, but little research has been conducted to examine how this phenomenon affects individuals’ self-concept. By employing the social adaptation hypothesis of female self-objectification as a theoretical framework, the present research firstly investigated how visual wealth exposure on social media contributed to the self-objectification of women, and the downstream consequences linked to the self-objectification. To this end, three preregistered online studies were conducted. By adopting an online questionnaire survey, Study 1 found that there was a significantly positive relationship between visual wealth exposure and self-objectification, and economic inequality perception and competence perception played a chain-mediating role between them. By randomly assigning participants to the wealth-relevant image exposure condition or the natural scenery image exposure condition, Study 2 conceptually replicated the findings of Study 1. The following Study 3 further found that, participants exposed to wealth-relevant images displayed more preferences for an appearance-dominated job (working as a network entertainment-anchor) than those exposed to natural scenery images, and self-objectification mediated the effect of exposure condition on career choice. The current research provided additional empirical supports for the social adaptation hypothesis of female self-objectification, and also deepened our understanding of the relationship between social media use and the self-objectification of women.

Universal scale laws for colors and patterns in imagery

Distribution of colors and patterns in images is observed through cascades that adjust spatial resolution and dynamics. Cascades of colors reveal the emergent universal property that Fully Colored Images (FCIs) of natural scenes adhere to the debated continuous linear log-scale law (slope −2.00±0.01) (L1). Cascades of discrete 2×2 patterns are derived from pixel square reductions onto the seven unlabeled rotation-free textures (0000, 0001, 0011, 0012, 0101, 0102, 0123). They exhibit an unparalleled universal entropy maximum of 1.74±0.013 at some dynamics regardless of spatial scale (L2). Patterns also adhere to the Integral Fluctuation Theorem (1.00±0.01) (L3), pivotal in studies of chaotic systems. Images with fewer colors exhibit quadratic shift and bias from L1 and L3 but adhere to L2. Randomized Hilbert fractal FCIs better match the laws than basic-to-AI-based simulations. Those results are of interest in Neural Networks, out-of-equilibrium physics, and spectral imagery.

Deep Learning Techniques for Detecting and Segmenting Text in Natural Scene Images: Review

Text detection and segmentation in natural scene images is an active research problem in computer vision and document analysis. Unlike scanned documents, scene text exhibits significant diversity in appearance, orientation, scale, font, and lighting conditions. In this review, survey the current state-of-the-art in techniques and methodologies aimed at detecting and segmenting text regions from images of natural scenes are presented. Both traditional approaches using hand-crafted features as well as modern data-driven deep learning methods will be discussed. The review will analyze common datasets, evaluation protocols and metrics used for benchmarking. Limitations of existing methods and open challenges in handling multilingual text, curved text, and efficiency will be highlighted. Promising future directions towards robust and generalizable scene text extraction systems will be identified. In summary, the review will provide a comprehensive overview of the advances, remaining challenges and future opportunities in developing automated systems for detecting and segmenting text in unconstrained natural images.

E2SCNet: Efficient Multiobjective Evolutionary Automatic Search for Remote Sensing Image Scene Classification Network Architecture.

Remote sensing image scene classification methods based on deep learning have been widely studied and discussed. However, most of the network architectures are directly reliant on natural image processing methods and are fixed. A few studies have focused on automatic search mechanisms, but they cannot weigh the interpretation accuracy and the parameter quantity for practical application. As a result, automatic global search methods based on multiobjective evolutionary computation have more advantages. However, in the ranking process, the network individuals with large parameter quantities are easy to eliminate, but a higher accuracy may be obtained after full training. In addition, evolutionary neural architecture search methods often take several days. In this article, in order to solve the above concerns, we propose an efficient multiobjective evolutionary automatic search framework for remote sensing image scene classification deep learning network architectures (E2SCNet). In E2SCNet, eight kinds of lightweight operators are used to build a diversified search space, and the coding connection mode is flexible. In the search process, a large model retention mechanism is implemented through two-step multiobjective modeling and evolutionary search, where one step involves the "parameter quantity and accuracy," and the other step involves the "parameter quantity and accuracy growth quantity." Moreover, a super network is constructed to share the weight in the process of individual network evaluation and promote the search speed. The effectiveness of E2SCNet is proven by comparison with several networks designed by human experts and networks obtained by gradient and evolutionary computing-based search methods.

MODIS daily cloud-gap-filled fractional snow cover dataset of the Asian Water Tower region (2000–2022)

Abstract. Accurate long-term daily cloud-gap-filled fractional snow cover products are essential for climate change and snow hydrological studies in the Asian Water Tower (AWT) region, but existing Moderate Resolution Imaging Spectroradiometer (MODIS) snow cover products are not sufficient. In this study, the multiple-endmember spectral mixture analysis algorithm based on automatic endmember extraction (MESMA-AGE) and the multistep spatiotemporal interpolation algorithm (MSTI) are used to produce the MODIS daily cloud-gap-filled fractional snow cover product over the AWT region (AWT MODIS FSC). The AWT MODIS FSC products have a spatial resolution of 0.005° and span from 2000 to 2022. The 2745 scenes of Landsat-8 images are used for the areal-scale accuracy assessment. The fractional snow cover accuracy metrics, including the coefficient of determination (R2), root mean squared error (RMSE) and mean absolute error (MAE), are 0.80, 0.16 and 0.10, respectively. The binarized identification accuracy metrics, including overall accuracy (OA), producer's accuracy (PA) and user's accuracy (UA), are 95.17 %, 97.34 % and 97.59 %, respectively. Snow depth data observed at 175 meteorological stations are used to evaluate accuracy at the point scale, yielding the following accuracy metrics: an OA of 93.26 %, a PA of 84.41 %, a UA of 82.14 % and a Cohen kappa (CK) value of 0.79. Snow depth observations from meteorological stations are also used to assess the fractional snow cover resulting from different weather conditions, with an OA of 95.36 % (88.96 %), a PA of 87.75 % (82.26 %), a UA of 86.86 % (78.86 %) and a CK of 0.84 (0.72) under the MODIS clear-sky observations (spatiotemporal reconstruction based on the MSTI algorithm). The AWT MODIS FSC product can provide quantitative spatial distribution information on snowpacks for mountain hydrological models, land surface models and numerical weather prediction in the Asian Water Tower region. This dataset is freely available from the National Tibetan Plateau Data Center at https://doi.org/10.11888/Cryos.tpdc.272503 (Jiang et al., 2022) or from the Zenodo platform at https://doi.org/10.5281/zenodo.10005826 (Jiang et al., 2023a).

FLsM: Fuzzy Localization of Image Scenes Based on Large Models

This article primarily focuses on the study of image-based localization technology. While traditional methods have made significant advancements in technology and applications, the emerging field of visual image-based localization technology demonstrates tremendous potential for research. Deep learning has exhibited a strong performance in image processing, particularly in developing visual navigation and localization techniques using large-scale visual models. This paper introduces a sophisticated scene image localization technique based on large models in a vast spatial sample environment. The study involved training convolutional neural networks using millions of geographically labeled images, extracting image position information using large model algorithms, and collecting sample data under various conditions in elastic scene space. Through visual computation, the shooting position of photos was inferred to obtain the approximate position information of users. This method utilizes geographic location information to classify images and combines it with landmarks, natural features, and architectural styles to determine their locations. The experimental results show variations in positioning accuracy among different models, with the most optimal model obtained through training on a large-scale dataset. They also indicate that the positioning error in urban street-based images is relatively small, whereas the positioning effect in outdoor and local scenes, especially in large-scale spatial environments, is limited. This suggests that the location information of users can be effectively determined through the utilization of geographic data, to classify images and incorporate landmarks, natural features, and architectural styles. The study’s experimentation indicates the variation in positioning accuracy among different models, highlighting the significance of training on a large-scale dataset for optimal results. Furthermore, it highlights the contrasting impact on urban street-based images versus outdoor and local scenes in large-scale spatial environments.

Densely integrated multi-branch attentive net for image dehazing

Single image dehazing is a fundamental but challenging task in image processing. Various deep learning-based methods have achieved great dehazing performance. However, there are still hazy residues, even color distortion and texture loss when removing haze from complex outdoor images in dense hazy scenes. A densely integrated multi-branch attentive net for image dehazing is proposed in the paper to address the above problems. The network includes a multi-scale feature extraction module and a dense-feature fusion module. The multi-scale feature extraction module adopts a multi-branch structure composed of residual channel attention blocks, which can expand the receptive field and filter the extracted features of diverse scales by weighting for fusion. It raises network learning accuracy. The dense-feature fusion module contains a multi-level feature fusion module for front and back layers, a color information renovation module, and a feature enhancement module. It achieves dynamically adjusting the channel weights of features at diverse scales, learning rich context information and suppressing redundant information, and compensating for the lack of color and texture information. The dense-feature fusion module bolsters the generalization capability of the network. The proposed method achieves superior objective and subjective evaluation results via quantitative and qualitative experiments on synthetic hazy images and natural hazy images, with better generalization ability and dehazing effect than the current SOTA dehazing algorithms, and effectively ameliorates the color distortion and incomplete dehazing.

Dehaze-UNet: A Lightweight Network Based on UNet for Single-Image Dehazing

Numerous extant image dehazing methods based on learning improve performance by increasing the depth or width, the size of the convolution kernel, or using the Transformer structure. However, this will inevitably introduce many parameters and increase the computational overhead. Therefore, we propose a lightweight dehazing framework: Dehaze-UNet, which has excellent dehazing performance and very low computational overhead to be suitable for terminal deployment. To allow Dehaze-UNet to aggregate the features of haze, we design a LAYER module. This module mainly aggregates the haze features of different hazy images through the batch normalization layer, so that Dehaze-UNet can pay more attention to haze. Furthermore, we revisit the use of the physical model in the network. We design an ASMFUN module to operate the feature map of the network, allowing the network to better understand the generation and removal of haze and learn prior knowledge to improve the network’s generalization to real hazy scenes. Extensive experimental results indicate that the lightweight Dehaze-UNet outperforms state-of-the-art methods, especially for hazy images of real scenes.

A Robust CoS-PVNet Pose Estimation Network in Complex Scenarios

Object 6D pose estimation, as a key technology in applications such as augmented reality (AR), virtual reality (VR), robotics, and autonomous driving, requires the prediction of the 3D position and 3D pose of objects robustly from complex scene images. However, complex environmental factors such as occlusion, noise, weak texture, and lighting changes may affect the accuracy and robustness of object 6D pose estimation. We propose a robust CoS-PVNet (complex scenarios pixel-wise voting network) pose estimation network for complex scenes. By adding a pixel-weight layer based on the PVNet network, more accurate pixel point vectors are selected, and dilated convolution and adaptive weighting strategies are used to capture local and global contextual information of the input feature map. At the same time, the perspective-n-point localization algorithm is used to accurately locate 2D key points to solve the pose of 6D objects, and then, the transformation relationship matrix of 6D pose projection is solved. The research results indicate that on the LineMod and Occlusion LineMod datasets, CoS-PVNet has high accuracy and can achieve stable and robust 6D pose estimation even in complex scenes.

Real Time Object Detection using TensorFlow

In recent years, deep learning has been used in image classification, object tracking, action recognition and scene labeling. Traditionally, Image Processing techniques were used to solve any Computer Vision problems occurred in an artificial intelligence system. However, in real-time identification, image processing cannot be used. This is where Deep Learning concepts are applied. We built a simple Convolutional Neural Network for object detection. The model is trained and multiple test cases are implemented in the TensorFlow environment so as to obtain accurate results

High-level vision task-driven infrared and visible image fusion approach: Progressive Semantic Enhancement based Multi-scale Cross-modality Interactive Network

Infrared-visible image fusion aims to merge data for a more accurate scene representation. Current research focuses mainly on enhancing visual appeal rather than improving performance in high-level vision tasks. To address this gap, we propose the Semantic Enhanced Multi-scale Cross-modality Interactive Image Fusion Network (SeMIFusion). Initially, Multi-scale Cross-modality Feature Fusion (MCFF) module is devised to extract shallow and deep features across different modalities. During feature extraction, Texture Enhancer (TE) and Semantify Enhancer (SE) blocks capture diverse hierarchical features across multi-scale layers, seamlessly integrating into the Semantic Feature Integration (SFI) module for profound semantic information extraction. Furthermore, an Image Scene Reconstruction (ISR) module maintains original image details in fused features, ensuring image fidelity. Additionally, incorporating a visual preservation guiding mask prioritizes retaining visual quality during reconstruction, preventing degradation. Extensive experiments demonstrate our method’s superiority in preserving visual effects and texture details, especially in high-level vision tasks. The source code is available at https://github.com/yyzzttkkjj/SeMIFusion.

Unsupervised Scene Image Text Segmentation Based on Improved CycleGAN

Scene image text segmentation is an important task in computer vision, but the complexity and diversity of backgrounds make it challenging. All supervised image segmentation tasks require paired semantic label data to ensure the accuracy of segmentation, but semantic labels are often difficult to obtain. To solve this problem, we propose an unsupervised scene image text segmentation model based on the image style transfer model cyclic uniform Generation Adversarial network (CycleGAN), which is trained by partial unpaired label data. Text segmentation is achieved by converting a complex background to a simple background. Since the images generated by CycleGAN cannot retain the details of the text content, we also introduced the Atrous spatial Pyramid pool module (ASPP) to obtain the features of the text from multiple scales. The resulting image quality is improved. The proposed method is verified by experiments on a synthetic data set, the IIIT 5k word data set and the MACT data set, which effectively segments the text and preserves the details of the text content.

An edge feature-based external calibration of monocular camera and LiDAR using in-orbit data

Multimodal perception represents a vital approach for observing non-cooperative targets in space. The accurate determination of the extrinsic parameters for monocular camera and LiDAR systems serves as the cornerstone for achieving effective space multimodal perception. However, due to factors like launch and operation, these extrinsic parameters can experience drift, thereby compromising the overall performance of the multimodal perception system. To tackle this challenge, this study introduces a novel approach: an edge feature-based external calibration method for monocular camera and LiDAR using in-orbit data. The proposed methodology involves several steps. Firstly, edges are extracted from the image, followed by distance transformation based on the edge image. The subsequent integration of the edge image and distance image yields the search scene image. Then, edge feature is extracted from the point cloud data. Finally, after constructing the objective function, genetic algorithms are used to determine precise external parameters. Simulation results underscore the effectiveness of the proposed algorithm, affirming its ability to achieve high-precision calibration results for external parameters even within intricate spatial environments.

FPS-U2Net: Combining U2Net and multi-level aggregation architecture for fire point segmentation in remote sensing images

Traditional methods for fire point segmentation (FPS) in satellite remote sensing images (RSIs) overly rely on threshold judgment, which are greatly affected by factors such as regional time and show poor generalization. Besides, due to the difference between natural scene images (NSIs) and RSIs, directly apply NSIs-based deep learning methods to forest fire RSIs without any modification fails to achieve satisfactory results. To address these issues, first, we construct a Landsat8 RSI-FPS dataset covering different years, seasons and regions. Then, for the first time, we apply salient object detection (SOD) to FPS in forest fire monitoring and propose a novel network FPS-U2Net to improve the performance of FPS. FPS-U2Net is based on U2Netp (a lightweight U2Net), to make better use of the multi-level features from adjacent encoders, we propose multi-level aggregation module (MAM), which is placed between the encoder and decoder at the same stage to aggregate the adjacent multi-scale features and capture richer contextual information. To make up for the weakness of BCE loss, we employ the hybrid loss, BCE + IoU, for the training of the network, which can guide the network learn the salient information from pixel and map levels. Extensive experiments on three datasets demonstrate that our FPS-U2Net significantly outperforms the state-of-the-art semantic segmentation and SOD methods. FPS-U2Net can accurately segment fire regions and predict clear local details.

Remote intelligent perception system for multi-object detection.

During the last few years, a heightened interest has been shown in classifying scene images depicting diverse robotic environments. The surge in interest can be attributed to significant improvements in visual sensor technology, which has enhanced image analysis capabilities. Advances in vision technology have a major impact on the areas of multiple object detection and scene understanding. These tasks are an integral part of a variety of technologies, including integrating scenes in augmented reality, facilitating robot navigation, enabling autonomous driving systems, and improving applications in tourist information. Despite significant strides in visual interpretation, numerous challenges persist, encompassing semantic understanding, occlusion, orientation, insufficient availability of labeled data, uneven illumination including shadows and lighting, variation in direction, and object size and changing background. To overcome these challenges, we proposed an innovative scene recognition framework, which proved to be highly effective and yielded remarkable results. First, we perform preprocessing using kernel convolution on scene data. Second, we perform semantic segmentation using UNet segmentation. Then, we extract features from these segmented data using discrete wavelet transform (DWT), Sobel and Laplacian, and textual (local binary pattern analysis). To recognize the object, we have used deep belief network and then find the object-to-object relation. Finally, AlexNet is used to assign the relevant labels to the scene based on recognized objects in the image. The performance of the proposed system was validated using three standard datasets: PASCALVOC-12, Cityscapes, and Caltech 101. The accuracy attained on the PASCALVOC-12 dataset exceeds 96% while achieving a rate of 95.90% on the Cityscapes dataset. Furthermore, the model demonstrates a commendable accuracy of 92.2% on the Caltech 101 dataset. This model showcases noteworthy advancements beyond the capabilities of current models.