Complex Scene Images Research Articles

Object-oriented U-GCN for open-pit mining extraction from high spatial resolution remote-sensing images of complex scenes

ABSTRACT Precise extraction of open-pit mines is crucial for resource management and ecological and environmental dynamic monitoring. Current methods for extracting open-pit mines encounter challenges such as low accuracy and difficulty detecting complex scenes of open-pit mining. To address these issues, this paper proposes an object-oriented intelligent extraction method for complex mining scenes using Gaofen-2(GF-2) high-resolution remote-sensing images, which expresses the pixel-level features at the object level and utilizes the unique feature propagation and aggregation capabilities of the graph structure for the extraction of the mines. First, an object-oriented feature expression strategy is introduced to express multi-level pixel-level features as object-level features by constructing objects with appropriate multi-resolution segmentation parameters. This approach effectively reduces the impact of isolated pixel noise and outliers on classification results. Second, this paper proposes a U-GCN-based open-pit mining extraction method that combines the powerful multi-level feature extraction capabilities of U-GCN to propagate and aggregate information in a graph structure, effectively modelling spatial relationships between different objects. This method achieves high-precision extraction of open-pit mining areas. In experiments conducted on two study areas of varying scales, the F1 scores for open-pit mine extraction reached 93.32% and 83.06%. Comparative experiments demonstrate that the proposed object-oriented U-GCN method performs superiorly in terms of accuracy, stability and robustness across mining scenes with different levels of complexity. The proposed open-pit mine extraction method offers new insights and methodologies for current extraction practices.

Real-Time Semantic Segmentation Algorithm for Street Scenes Based on Attention Mechanism and Feature Fusion

In computer vision, the task of semantic segmentation is crucial for applications such as autonomous driving and intelligent surveillance. However, achieving a balance between real-time performance and segmentation accuracy remains a significant challenge. Although Fast-SCNN is favored for its efficiency and low computational complexity, it still faces difficulties when handling complex street scene images. To address this issue, this paper presents an improved Fast-SCNN, aiming to enhance the accuracy and efficiency of semantic segmentation by incorporating a novel attention mechanism and an enhanced feature extraction module. Firstly, the integrated SimAM (Simple, Parameter-Free Attention Module) increases the network’s sensitivity to critical regions of the image and effectively adjusts the feature space weights across channels. Additionally, the refined pyramid pooling module in the global feature extraction module captures a broader range of contextual information through refined pooling levels. During the feature fusion stage, the introduction of an enhanced DAB (Depthwise Asymmetric Bottleneck) block and SE (Squeeze-and-Excitation) attention optimizes the network’s ability to process multi-scale information. Furthermore, the classifier module is extended by incorporating deeper convolutions and more complex convolutional structures, leading to a further improvement in model performance. These enhancements significantly improve the model’s ability to capture details and overall segmentation performance. Experimental results demonstrate that the proposed method excels in processing complex street scene images, achieving a mean Intersection over Union (mIoU) of 71.7% and 69.4% on the Cityscapes and CamVid datasets, respectively, while maintaining inference speeds of 81.4 fps and 113.6 fps. These results indicate that the proposed model effectively improves segmentation quality in complex street scenes while ensuring real-time processing capabilities.

A Robust CoS-PVNet Pose Estimation Network in Complex Scenarios

Object 6D pose estimation, as a key technology in applications such as augmented reality (AR), virtual reality (VR), robotics, and autonomous driving, requires the prediction of the 3D position and 3D pose of objects robustly from complex scene images. However, complex environmental factors such as occlusion, noise, weak texture, and lighting changes may affect the accuracy and robustness of object 6D pose estimation. We propose a robust CoS-PVNet (complex scenarios pixel-wise voting network) pose estimation network for complex scenes. By adding a pixel-weight layer based on the PVNet network, more accurate pixel point vectors are selected, and dilated convolution and adaptive weighting strategies are used to capture local and global contextual information of the input feature map. At the same time, the perspective-n-point localization algorithm is used to accurately locate 2D key points to solve the pose of 6D objects, and then, the transformation relationship matrix of 6D pose projection is solved. The research results indicate that on the LineMod and Occlusion LineMod datasets, CoS-PVNet has high accuracy and can achieve stable and robust 6D pose estimation even in complex scenes.

TSD: Random feature query design for transformer-based shrimp detector

Shrimp farming plays a vital role in ensuring food security, income generation, and employment opportunities for millions of people globally. One promising approach to enhance the efficiency of shrimp farming is through shrimp image detection, which uses computer vision to identify shrimp in aquaculture ponds. However, detecting shrimp in complex environments can be challenging, as light conditions and water turbidity can interfere with existing detection models. The current shrimp detection methods suffer from issues such as low detection accuracy and limited robustness. Therefore, we presents a Transformer-based Shrimp Detector (TSD) framework for detecting shrimp in images of complex scenes. The algorithm employs a Convolutional Neural Network to extract image features, which are then fed into a transformer encoder–decoder. We also introduce an innovative object query setting method in the decoder that uses a random feature query. To predict the detection results, a feed-forward neural network is used, and two parts in the matching loss are improved. We construct a dataset of images containing diverse shrimp farming environments, and provide box-level annotations. The experimental results demonstrate that the proposed method achieves a detection accuracy of 82.7% Average Precision, surpassing mainstream object detection models.

Learning Multi-Object Positional Relationships via Emergent Communication

The study of emergent communication has been dedicated to interactive artificial intelligence. While existing work focuses on communication about single objects or complex image scenes, we argue that communicating relationships between multiple objects is important in more realistic tasks, but understudied. In this paper, we try to fill this gap and focus on emergent communication about positional relationships between two objects. We train agents in the referential game where observations contain two objects, and find that generalization is the major problem when the positional relationship is involved. The key factor affecting the generalization ability of the emergent language is the input variation between Speaker and Listener, which is realized by a random image generator in our work. Further, we find that the learned language can generalize well in a new multi-step MDP task where the positional relationship describes the goal, and performs better than raw-pixel images as well as pre-trained image features, verifying the strong generalization ability of discrete sequences. We also show that language transfer from the referential game performs better in the new task than learning language directly in this task, implying the potential benefits of pre-training in referential games. All in all, our experiments demonstrate the viability and merit of having agents learn to communicate positional relationships between multiple objects through emergent communication.

SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation

Despite significant progress in Text-to-Image (T2I) generative models, even lengthy and complex text descriptions still struggle to convey detailed controls. In contrast, Layout-to-Image (L2I) generation, aiming to generate realistic and complex scene images from user-specified layouts, has risen to prominence. However, existing methods transform layout information into tokens or RGB images for conditional control in the generative process, leading to insufficient spatial and semantic controllability of individual instances. To address these limitations, we propose a novel Spatial-Semantic Map Guided (SSMG) diffusion model that adopts the feature map, derived from the layout, as guidance. Owing to rich spatial and semantic information encapsulated in well-designed feature maps, SSMG achieves superior generation quality with sufficient spatial and semantic controllability compared to previous works. Additionally, we propose the Relation-Sensitive Attention (RSA) and Location-Sensitive Attention (LSA) mechanisms. The former aims to model the relationships among multiple objects within scenes while the latter is designed to heighten the model's sensitivity to the spatial information embedded in the guidance. Extensive experiments demonstrate that SSMG achieves highly promising results, setting a new state-of-the-art across a range of metrics encompassing fidelity, diversity, and controllability.

Quasi-Dense Matching for Oblique Stereo Images through Semantic Segmentation and Local Feature Enhancement

This paper proposes a quasi-dense feature matching algorithm that combines image semantic segmentation and local feature enhancement networks to address the problem of the poor matching of image features because of complex distortions, considerable occlusions, and a lack of texture on large oblique stereo images. First, a small amount of typical complex scene data are used to train the VGG16-UNet, followed by completing the semantic segmentation of multiplanar scenes across large oblique images. Subsequently, the prediction results of the segmentation are subjected to local adaptive optimization to obtain high-precision semantic segmentation results for each planar scene. Afterward, the LoFTR (Local Feature Matching with Transformers) strategy is used for scene matching, enabling enhanced matching for regions with poor local texture in the corresponding planes. The proposed method was tested on low-altitude large baseline stereo images of complex scenes and compared with five classical matching methods. Results reveal that the proposed method exhibits considerable advantages in terms of the number of correct matches, correct rate of matches, matching accuracy, and spatial distribution of corresponding points. Moreover, it is well-suitable for quasi-dense matching tasks of large baseline stereo images in complex scenes with considerable viewpoint variations.

Oriented Ship Detection Based on Soft Thresholding and Context Information in SAR Images of Complex Scenes

Oriented Ship Detection Based on Soft Thresholding and Context Information in SAR Images of Complex Scenes

SiamFCA: A new fish single object tracking method based on siamese network with coordinate attention in aquaculture

In aquaculture, tracking fish helps to obtain information about their growth, behavior, location and health, etc. However, there are some problems in the process of fish tracking. Firstly, to solve the noise problem of underwater images, we use the contrast limited adaptive histogram equalization (CLAHE) to carry out local histogram equalization in each small block, which increases the image contrast while retaining more details. This operation enhances the contrast of the image while suppressing noise to solve the problem of noisy images. Then, to solve the problems of similarity interference, occlusion and scale variation among fish, we propose a new method for fish single object tracking in aquaculture environments, called SiamFCA, by combining the coordinated attention mechanism (CA). The method is based on the siamese network framework and uses a modified AlexNet to extract the depth features of image pairs. And through the coordinate attention mechanism, the model can learn the relationship between different objects more accurately, which can improve the accuracy and robustness of the model when dealing with complex image scenes. Finally, the correlation operation between classification and regression is performed based on the region proposal network (RPN). By classifying, region suggestions can be labeled as positive or negative samples, and regression can calibrate the position and size of the region proposal to match targets more accurately. The experimental results show that the precision and success rate of the algorithm are 84.2 % and 62.4 %, respectively, which are better than the tracking methods such as SiamRPN and DaSiamRPN, etc. The SiamFCA runs at 83 fps, which meets the demand of real-time fish tracking in aquaculture.

Dynamic receptive field adaptation for scene text recognition

Scene text recognition methods of the Encoder–Decoder framework, generally assume that the proportion of characters in the same text instance are basically the same. However, this assumption does not always hold in the context of complex scene images. For adaptively revising the receptive field according to the different font in the scene text image, we propose a Dynamic Receptive Field Adaption Framework which consists of Memory Attention (MA) module and Dynamic Feature Adaptive (DFA) module. MA percepts the historical location information to adapt to the change of character position in the decoder. DFA selects the most distinguishing features from feature maps of different levels dynamically. Additionally, MA and DFA can be easily extended to the existing attention-based and transformer-based text recognition methods to improve their performance. With extension experiments on public benchmark datasets, including IIIT-5K, SVT, SVTP, CUTE80, RECTS, LSVT, and RCTW, our method has shown effectiveness and robustness.

Natureness‐preserving tone mapping operator based on improved guided filter and adaptive Gamma curve

AbstractMapping high dynamic range (HDR) images to low dynamic range (LDR) display devices while performing enhancement processing is a challenging task. To address the issues of chromatic drift, the imbalance between preserving naturalness and enhancing details during the mapping process, this paper proposed a tone mapping operator (TMO) based on an improved guided filter with multi‐layered processing. Firstly, the operator tackled the problem of insufficient extraction of detail layer information in complex scene images by redefining the formula of the guided filter and introducing adaptive regularization parameters. This enhancement enabled more effective extraction of details compared to traditional fixed‐parameter guided filters. Secondly, the algorithm employed a centre‐surround approach to subdivide and processed the detail layers individually, effectively suppressing halos and artefacts. Thirdly, image entropy was utilized to guide the fusion and enhancement processing of the detail layers, preventing excessive or insufficient enhancement. Finally, this paper proposed an adaptive gamma correction algorithm and a chromatic consistency algorithm to improve overall contrast and hue saturation. The results demonstrate convincing performance in terms of subjective visual evaluation and image quality assessment.

Textual Characters Detection in Complex Scene Images based on Bradley Thresholding Technique in Compare with Statistical Thresholding Technique

Textual Characters Detection in Complex Scene Images based on Bradley Thresholding Technique in Compare with Statistical Thresholding Technique

Natural scene reconstruction from fMRI signals using generative latent diffusion

In neural decoding research, one of the most intriguing topics is the reconstruction of perceived natural images based on fMRI signals. Previous studies have succeeded in re-creating different aspects of the visuals, such as low-level properties (shape, texture, layout) or high-level features (category of objects, descriptive semantics of scenes) but have typically failed to reconstruct these properties together for complex scene images. Generative AI has recently made a leap forward with latent diffusion models capable of generating high-complexity images. Here, we investigate how to take advantage of this innovative technology for brain decoding. We present a two-stage scene reconstruction framework called “Brain-Diffuser”. In the first stage, starting from fMRI signals, we reconstruct images that capture low-level properties and overall layout using a VDVAE (Very Deep Variational Autoencoder) model. In the second stage, we use the image-to-image framework of a latent diffusion model (Versatile Diffusion) conditioned on predicted multimodal (text and visual) features, to generate final reconstructed images. On the publicly available Natural Scenes Dataset benchmark, our method outperforms previous models both qualitatively and quantitatively. When applied to synthetic fMRI patterns generated from individual ROI (region-of-interest) masks, our trained model creates compelling “ROI-optimal” scenes consistent with neuroscientific knowledge. Thus, the proposed methodology can have an impact on both applied (e.g. brain–computer interface) and fundamental neuroscience.

Hierarchical attention aggregation with multi-resolution feature learning for GAN-based underwater image enhancement

In recent years, underwater image enhancement and restoration technologies have become increasingly important in order to optimize the efficiency of maritime operations and promote the automatic machine learning of the maritime industry. A new hierarchical attention aggregation with multi-resolution feature learning for GAN-based underwater image enhancement is proposed to address the problems of color bias, underexposure, and blurring in underwater images. The proposed method consists of a generator and a discriminator. Specifically, the generator includes an encoder, a bottleneck layer, and a decoder. Generator introduces inter-block serial connections for better adaptation to complex image scenes and task requirements, and parallel connections to extract multi-level features and enhance the expressive capacity of the network. To extract semantic and contextual information, hierarchical attention dense aggregation is designed in the encoder, which includes multi-scale feature hierarchy and dense feature hierarchy. Additionally, a multi-scale spatial attention mechanism is designed in the bottleneck layer to handle variations in underwater image scenes. In the decoder, the feature channel layer is emphasized, and a multi-channel attention mechanism is proposed to restore the multi-resolution channel features to a three-channel enhanced image. Furthermore, the angular loss function is introduced as additional supervision, which improves the similarity between the generated and original images, increases image clarity, and reduces color bias. Meanwhile, we employ the patch discriminator to enhance machine vision. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art methods.

Multi-Scale Ship Detection Algorithm Based on YOLOv7 for Complex Scene SAR Images

Recently, deep learning techniques have been extensively used to detect ships in synthetic aperture radar (SAR) images. The majority of modern algorithms can achieve successful ship detection outcomes when working with multiple-scale ships on a large sea surface. However, there are still issues, such as missed detection and incorrect identification when performing multi-scale ship object detection operations in SAR images of complex scenes. To solve these problems, this paper proposes a complex scenes multi-scale ship detection model, according to YOLOv7, called CSD-YOLO. First, this paper suggests an SAS-FPN module that combines atrous spatial pyramid pooling and shuffle attention, allowing the model to focus on important information and ignore irrelevant information, reduce the feature loss of small ships, and simultaneously fuse the feature maps of ship targets on various SAR image scales, thereby improving detection accuracy and the model’s capacity to detect objects at several scales. The model’s optimization is then improved with the aid of the SIoU loss function. Finally, thorough tests on the HRSID and SSDD datasets are presented to support our methodology. CSD-YOLO achieves better detection performance than the baseline YOLOv7, with a 98.01% detection accuracy, a 96.18% recall, and a mean average precision (mAP) of 98.60% on SSDD. In addition, in comparative experiments with other deep learning-based methods, in terms of overall performance, CSD-YOLO still performs better.

A Convolutional Recurrent Neural-Network-Based Machine Learning for Scene Text Recognition Application

Optical character recognition (OCR) is the process of acquiring text and layout information through analysis and recognition of text data image files. It is also a process to identify the geometric location and orientation of the texts and their symmetrical behavior. It usually consists of two steps: text detection and text recognition. Scene text recognition is a subfield of OCR that focuses on processing text in natural scenes, such as streets, billboards, license plates, etc. Unlike traditional document category photographs, it is a challenging task to use computer technology to locate and read text information in natural scenes. Imaging sequence recognition is a longstanding subject of research in the field of computer vision. Great progress has been made in this field; however, most models struggled to recognize text in images of complex scenes with high accuracy. This paper proposes a new pattern of text recognition based on the convolutional recurrent neural network (CRNN) as a solution to address this issue. It combines real-time scene text detection with differentiable binarization (DBNet) for text detection and segmentation, text direction classifier, and the Retinex algorithm for image enhancement. To evaluate the effectiveness of the proposed method, we performed experimental analysis of the proposed algorithm, and carried out simulation on complex scene image data based on existing literature data and also on several real datasets designed for a variety of nonstationary environments. Experimental results demonstrated that our proposed model performed better than the baseline methods on three benchmark datasets and achieved on-par performance with other approaches on existing datasets. This model can solve the problem that CRNN cannot identify text in complex and multi-oriented text scenes. Furthermore, it outperforms the original CRNN model with higher accuracy across a wider variety of application scenarios.

Fire and smoke precise detection method based on the attention mechanism and anchor-free mechanism

Substantial natural environmental damage and economic losses are caused by fire. For this problem, automatic fire-smoke detection and identification are needed. Fire-smoke detection methods based on vision still suffer from significant challenges that fail to balance model complexity and accuracy. We propose an improved YOLOv3 fire-smoke detection and identification method to address these problems and include a fire and smoke dataset. The neck module (1) adds an attention mechanism to enhance the ability to extract features from pictures, and (2) uses an anchor-free mechanism in the anchor box mechanism to solve the problem of significant variances in smoke texture, shape, and color in real applications, and (3) uses a lightweight backbone to reduce the model complexity. The proposed dataset is based on VOC, which contains images of complex scenes and high diversity. The dataset includes pictures that (1) combine fire with smoke, (2) only have smoke or fire objects, and (3) contain a single cloud object. The experimental results demonstrate that the method achieves 50.8 AP, which outperforms the suboptimal method by 3.8. Moreover, the inference speed of our method is 13% faster on the GPU than the suboptimal method.

Refined edge detection with cascaded and high-resolution convolutional network

Edge detection is represented as one of the most challenging tasks in computer vision, due to the complexity of detecting the edges or boundaries in real-world images that contains objects of different types and scales like trees, building as well as various backgrounds. Edge detection is represented also as a key task for many computer vision applications. Using a set of backbones as well as attention modules, deep-learning-based methods improved the detection of edges compared with traditional methods like Sobel or Canny. However, images of complex scenes still represent a challenge for these methods. Also, the detected edges using the existing approaches suffer from non-refined results with erroneous edges. In this paper, we attempted to overcome these challenges for refined edge detection using a cascaded and high-resolution network named (CHRNet). By maintaining the high resolution of edges during the training process, and conserving the resolution of the edge image during the network stage, sub-blocks are connected at every stage with the output of the previous layer. Also, after each layer, we use batch normalization layer with an active affine parameter as an erosion operation for the homogeneous region in the image. The proposed method is evaluated using the most challenging datasets including BSDS500, NYUD, and Multicue. The obtained results outperform the designed edge detection networks in terms of performance metrics and quality of output images.The code is available at: https://github.com/elharroussomar/chrnet/

Image Captioning Using Detectors and Swarm Based Learning Approach for Word Embedding Vectors

IC (Image Captioning) is a crucial part of Visual Data Processing and aims at understanding for providing captions that verbalize an image’s important elements. However, in existing works, because of the complexity in images, neglecting major relation between the object in an image, poor quality image, labelling it remains a big problem for researchers. Hence, the main objective of this work attempts to overcome these challenges by proposing a novel framework for IC. So in this research work the main contribution deals with the framework consists of three phases that is image understanding, textual understanding and decoding. Initially, the image understanding phase is initiated with image pre-processing to enhance image quality. Thereafter, object has been detected using IYV3MMDs (Improved YoloV3 Multishot Multibox Detectors) in order to relate the interrelation between the image and the object, and then it is followed by MBFOCNNs (Modified Bacterial Foraging Optimization in Convolution Neural Networks), which encodes and provides final feature vectors. Secondly, the textual understanding phase is performed based on an image which is initiated with preprocessing of text where unwanted words, phrases, punctuations are removed in order to provide a healthy text. It is then followed by MGloVEs (Modified Global Vectors for Word Representation), which provides a word embedding of features with the highest priority towards the object present in an image. Finally, the decoding phase has been performed, which decodes the image whether it may be a normal or complex scene image and provides an accurate text by its learning ability using MDAA (Modified Deliberate Adaptive Attention). The experimental outcome of this work shows better accuracy of shows 96.24% when compared to existing and similar methods while generating captions for images.

Building Extraction From High Spatial Resolution Remote Sensing Images of Complex Scenes by Combining Region-Line Feature Fusion and OCNN

Building extraction from remote sensing imagery has been a research hotspot for some time with the advancement of AI in remote sensing. However, the edges of buildings extracted using existing techniques are commonly broken and inaccurate for the complex scenes in suburban and rural areas. This study proposes a framework for extracting structures by combining region-line feature fusion with object-based convolutional neural networks to solve this problem. First, a building edge detection network known as the Multichannel Attention-based Dense Extreme Inception Network for Edge Detection (MA-DexiNed) is constructed, which is considered more accurate for building edge extraction in complicated image scenes. Second, the probability map of the building edges obtained by MA-DexiNed is refined. According to rule judgment, breakpoints are linked by an edge thinning connection algorithm to obtain single-pixel, contiguous building line features. Third, the geometric boundaries of buildings are obtained by combining region attributes derived by unsupervised image segmentation and line features obtained from deep learning supervised segmentation. Finally, the pretrained AlexNet is employed to identify the class characteristics of buildings. The suggested framework was used for two GF-2 images and one Google Earth image from various regions and with numerous types of complicated scenes. The experimental findings demonstrated that this approach could extract more precise and complete building edges for complex image scenes compared with several existing methods. This advancement results from constrained regional image segmentation using deep semantic edge features. This methodology can offer a benchmark for subsequent building extraction tasks from high resolution imagery.