Scene Variations Research Articles

On the use of synthetic data for body detection in maritime search and rescue operations

Time is a critical factor in maritime Search And Rescue (SAR) missions, during which promptly locating survivors is paramount. Unmanned Aerial Vehicles (UAVs) are a useful tool with which to increase the success rate by rapidly identifying targets. While this task can be performed using other means, such as helicopters, the cost-effectiveness of UAVs makes them an effective choice. Moreover, these vehicles allow the easy integration of automatic systems that can be used to assist in the search process. Despite the impact of artificial intelligence on autonomous technology, there are still two major drawbacks to overcome: the need for sufficient training data to cover the wide variability of scenes that a UAV may encounter and the strong dependence of the generated models on the specific characteristics of the training samples. In this work, we address these challenges by proposing a novel approach that leverages computer-generated synthetic data alongside novel modifications to the You Only Look Once (YOLO) architecture that enhance its robustness, adaptability to new environments, and accuracy in detecting small targets. Our method introduces a new patch-sample extraction technique and task-specific data augmentation, ensuring robust performance across diverse weather conditions. The results demonstrate our proposal’s superiority, showing an average 28% relative improvement in mean Average Precision (mAP) over the best-performing state-of-the-art baseline under training conditions with sufficient real data, and a remarkable 218% improvement when real data is limited. The proposal also presents a favorable balance between efficiency, effectiveness, and resource requirements.

An Effective Yak Behavior Classification Model with Improved YOLO-Pose Network Using Yak Skeleton Key Points Images

Yak behavior is a valuable indicator of their welfare and health. Information about important statuses, including fattening, reproductive health, and diseases, can be reflected and monitored through several indicative behavior patterns. In this study, an improved YOLOv7-pose model was developed to detect six yak behavior patterns in real time using labeled yak key-point images. The model was trained using labeled key-point image data of six behavior patterns including walking, feeding, standing, lying, mounting, and eliminative behaviors collected from seventeen 18-month-old yaks for two weeks. There were another four YOLOv7-pose series models trained as comparison methods for yak behavior pattern detection. The improved YOLOv7-pose model achieved the best detection performance with precision, recall, mAP0.5, and mAP0.5:0.95 of 89.9%, 87.7%, 90.4%, and 76.7%, respectively. The limitation of this study is that the YOLOv7-pose model detected behaviors under complex conditions, such as scene variation, subtle leg postures, and different light conditions, with relatively lower precision, which impacts its detection performance. Future developments in yak behavior pattern detection will amplify the simple size of the dataset and will utilize data streams like optical and video streams for real-time yak monitoring. Additionally, the model will be deployed on edge computing devices for large-scale agricultural applications.

Semantic Segmentation of Satellite Images for Landslide Detection Using Foreground-Aware and Multi-Scale Convolutional Attention Mechanism.

Advancements in satellite and aerial imagery technology have made it easier to obtain high-resolution remote sensing images, leading to widespread research and applications in various fields. Remote sensing image semantic segmentation is a crucial task that provides semantic and localization information for target objects. In addition to the large-scale variation issues common in most semantic segmentation datasets, aerial images present unique challenges, including high background complexity and imbalanced foreground-background ratios. However, general semantic segmentation methods primarily address scale variations in natural scenes and often neglect the specific challenges in remote sensing images, such as inadequate foreground modeling. In this paper, we present a foreground-aware remote sensing semantic segmentation model. The model introduces a multi-scale convolutional attention mechanism and utilizes a feature pyramid network architecture to extract multi-scale features, addressing the multi-scale problem. Additionally, we introduce a Foreground-Scene Relation Module to mitigate false alarms. The model enhances the foreground features by modeling the relationship between the foreground and the scene. In the loss function, a Soft Focal Loss is employed to focus on foreground samples during training, alleviating the foreground-background imbalance issue. Experimental results indicate that our proposed method outperforms current state-of-the-art general semantic segmentation methods and transformer-based methods on the LS dataset benchmark.

Nighttime Pothole Detection: A Benchmark

In the field of computer vision, the detection of road potholes at night represents a critical challenge in enhancing the safety of intelligent transportation systems. Ensuring road safety is of paramount importance, particularly in promptly repairing pothole issues. These abrupt road depressions can easily lead to vehicle skidding, loss of control, and even traffic accidents, especially when water has pooled in or submerged the potholes. Therefore, the detection and recognition of road potholes can significantly reduce vehicle damage and the incidence of safety incidents. However, research on road pothole detection lacks high-quality annotated datasets, particularly under low-light conditions at night. To address this issue, this study introduces a novel Nighttime Pothole Dataset (NPD), independently collected and comprising 3831 images that capture diverse scene variations. The construction of this dataset aims to counteract the insufficiency of existing data resources and strives to provide a richer and more realistic benchmark. Additionally, we develop a baseline detector, termed WT-YOLOv8, for the proposed dataset, based on YOLOv8. We also evaluate the performance of the improved WT-YOLOv8 method and eight state-of-the-art object detection methods on the NPD and the COCO dataset. The experimental results on the NPD demonstrate that WT-YOLOv8 achieves a 2.3% improvement in mean Average Precision (mAP) over YOLOv8. In terms of the key metrics—AP@0.5 and AP@0.75—it shows enhancements of 1.5% and 2.8%, respectively, compared to YOLOv8. The experimental results provide valuable insights into each method’s strengths and weaknesses under low-light conditions. This analysis highlights the importance of a specialized dataset for nighttime pothole detection and shows variations in accuracy and robustness among methods, emphasizing the need for improved nighttime pothole detection techniques. The introduction of the NPD is expected to stimulate further research, encouraging the development of advanced algorithms for nighttime pothole detection, ultimately leading to more flexible and reliable road maintenance and road safety.

The visual statistical learning overcomes scene dissimilarity through an independent clustering process.

Contextual cueing is a phenomenon of visual statistical learning observed in visual search tasks. Previous research has found that the degree of deviation of items from its centroid, known as variability, determines the extent of generalization for that repeated scene. Introducing variability increases dissimilarity between multiple occurrences of the same repeated layout significantly. However, current theories do not explain the mechanisms that help to overcome this dissimilarity during contextual cue learning. We propose that the cognitive system initially abstracts specific scenes into scene layouts through an automatic clustering unrelated to specific repeated scenes, and subsequently uses these abstracted scene layouts for contextual cue learning. Experiment 1 indicates that introducing greater variability in search scenes leads to a hindering in the contextual cue learning. Experiment 2 further establishes that conducting extensive visual searches involving spatial variability in entirely novel scenes facilitates subsequent contextual cue learning involving corresponding scene variability, confirming that learning clustering knowledge precedes the contextual cue learning and is independent of specific repeated scenes. Overall, this study demonstrates the existence of multiple levels of learning in visual statistical learning, where item-level learning can serve as material for layout-level learning, and the generalization reflects the constraining role of item-level knowledge on layout-level knowledge.

SAM-driven MAE pre-training and background-aware meta-learning for unsupervised vehicle re-identification

Distinguishing identity-unrelated background information from discriminative identity information poses a challenge in unsupervised vehicle re-identification (Re-ID). Re-ID models suffer from varying degrees of background interference caused by continuous scene variations. The recently proposed segment anything model (SAM) has demonstrated exceptional performance in zero-shot segmentation tasks. The combination of SAM and vehicle Re-ID models can achieve efficient separation of vehicle identity and background information. This paper proposes a method that combines SAM-driven mask autoencoder (MAE) pre-training and background-aware meta-learning for unsupervised vehicle Re-ID. The method consists of three sub-modules. First, the segmentation capacity of SAM is utilized to separate the vehicle identity region from the background. SAM cannot be robustly employed in exceptional situations, such as those with ambiguity or occlusion. Thus, in vehicle Re-ID downstream tasks, a spatially-constrained vehicle background segmentation method is presented to obtain accurate background segmentation results. Second, SAM-driven MAE pre-training utilizes the aforementioned segmentation results to select patches belonging to the vehicle and to mask other patches, allowing MAE to learn identity-sensitive features in a self-supervised manner. Finally, we present a background-aware meta-learning method to fit varying degrees of background interference in different scenarios by combining different background region ratios. Our experiments demonstrate that the proposed method has state-of-the-art performance in reducing background interference variations.

Transformer-based berm detection for automated bulldozer safety in edge dumping

Ensuring the safety of automated bulldozers during edge dumping operations in open-pit mines requires accurate detection of the berm's edge. However, precisely detecting berm edges in open-pit mining scenes is challenging due to the presence of similar texture structures. To address this issue, this paper created a dataset for berm edge detection that covers multiple scenes and proposed the edge detection model with transformer (MEDTER). This model, based on a transformer with a two-stage detection structure, aims to restore thin edges. Additionally, the deep multilevel aggregation decoder enhances the information flow in the transformer to improve the sensitivity to detailed information. The BCE-Dice loss functions effectively handle the problem of pixel imbalance. Results demonstrate that the proposed model achieves significant improvements compared to others. Furthermore, the model exhibits excellent detection performance despite variations in scenes and detection distances, suggesting its practical value in ensuring the safety of automated bulldozers during edge dumping.

AEMNet: Unsupervised Video Anomaly Detection Method Based on Attention-Enhanced Memory Networks

Video anomaly detection has always been a challenging task in computer vision due to data imbalance and susceptibility to scene variations such as lighting and occlusions. In response to this challenge, this paper proposes an unsupervised video anomaly detection method based on an attention-enhanced memory network. The method utilizes a dual-stream network structure of autoencoders, enhancing the model’s learning ability for important features in appearance and motion by introducing coordinate attention mechanisms and variance attention mechanisms, emphasizing significant characteristics of static objects and rapidly moving regions. By adding memory modules to both the appearance and motion branches, the network structure’s memory information is reinforced, enabling it to capture long-term spatiotemporal dependencies in videos and thereby improving the accuracy of anomaly detection. Furthermore, by optimizing the network structure’s activation functions to handle negative inputs, it enhances its nonlinear modeling capabilities, enabling better adaptation to complex environments, including variations in lighting and occlusions, further improving the effectiveness of anomaly detection. The paper conducts comparative experiments and ablation studies using three public available datasets and various models. The results demonstrate that compared to baseline models, the AUC performance is improved by 3.9%, 4.7%, and 1.7% on UCSD Ped2, CHUK Avenue, and ShanghaiTech datasets, respectively. When compared with the other models, the average AUC performance is improved by 4.3%, 5.4%, and 6.2%, with an average improvement of 8.75% in the ERR metric, validating the effectiveness and adaptability of the proposed method. The code can be obtained at the following URL: https://github.com/AcademicWhite/AEMNet .

Cross-Domain Person Re-Identification Based on Feature Fusion Invariance

Cross-domain person re-identification is a technique for identifying the same individual across different cameras or environments that necessitates the overcoming of challenges posed by scene variations, which is a primary challenge in person re-identification and a bottleneck for its practical applications. In this paper, we learn the invariance model of cross-domain feature fusion in a labeled source domain and an unlabeled target domain. First, our method learns the global and local fusion features of a person in the source domain by means of supervised learning with no component label and only person identification and obtains the fusion features of the person in the source and target domains by means of unsupervised learning. Based on person fusion features, this paper introduces feature memory to store the fused target features and designs a cross-domain invariance loss function to improve the cross-domain adaptability of the person. Finally, this paper carries out cross-domain person re-identification verification experiments between the Market-1501 and DukeMTMC-reID datasets; the experimental results show that the proposed method achieves significant performance improvement in cross-domain person re-identification.

Teacher-Student Framework for Polyphonic Semi-supervised Sound Event Detection: Survey and Empirical Analysis

Polyphonic sound event detection refers to the task of automatically identifying sound events occurring simultaneously in an auditory scene. Due to the inherent complexity and variability of real-world auditory scenes, building robust detectors for polyphonic sound event detection poses a significant challenge. The task becomes further more challenging without sufficient annotated data to develop sound event detection systems under a supervised learning regime. In this paper, we explore the recent developments in polyphonic sound event detection, with a particular emphasis on the application of Teacher-Student techniques within the semi-supervised learning paradigm. Unlike previous works, we have consolidated and organized the fragmented literature on Teacher-Student techniques for polyphonic sound event detection. By examining the latest research, categorizing Teacher-Student approaches, and conducting an empirical study to assess the performance of each approach, this survey offers valuable insights and practical guidance for researchers and practitioners in the field. Our findings highlight the potential benefits of utilizing multiple learners, ensuring consistent predictions, and making thoughtful choices regarding perturbation strategies.

Meaning Visualization: A Saussurean Semiotic Analysis of the Film Merry Riana

The objective of this study is to determine the motivational messages that are sent in the movie "Merry Riana: Million Dollar Dream" to identify the inspirational messages. The semiotic theory developed by Ferdinand de Saussure is used in this study. Through the use of this theoretical method, the meaning of signified and signifiers that are present in the movie Merry Riana is investigated. Identification of signified and signifier aspects in the film Merry Riana is carried out through variations of scenes that appear in the film. Apart from that, the method used is a qualitative approach by collecting data through observation and interviews with several sources who have insight into the film or were inspired by the film Merry Riana. The informants who conducted this interview included general audiences, film academics, and the film community, totaling nine informants. The result of the research finds that the film Merry Riana can inspire its viewers, and there are many inspirational messages for the audience in the film Merry Riana.

Crowd Counting Based on Multiscale Spatial Guided Perception Aggregation Network.

Crowd counting has received extensive attention in the field of computer vision, and methods based on deep convolutional neural networks (CNNs) have made great progress in this task. However, challenges such as scale variation, nonuniform distribution, complex background, and occlusion in crowded scenes hinder the performance of these networks in crowd counting. In order to overcome these challenges, this article proposes a multiscale spatial guidance perception aggregation network (MGANet) to achieve efficient and accurate crowd counting. MGANet consists of three parts: multiscale feature extraction network (MFEN), spatial guidance network (SGN), and attention fusion network (AFN). Specifically, to alleviate the scale variation problem in crowded scenes, MFEN is introduced to enhance the scale adaptability and effectively capture multiscale features in scenes with drastic scale variation. To address the challenges of nonuniform distribution and complex background in population, an SGN is proposed. The SGN includes two parts: the spatial context network (SCN) and the guidance perception network (GPN). SCN is used to capture the detailed semantic information between the multiscale feature positions extracted by MFEN, and improve the ability of deep structured information exploration. At the same time, the dependence relationship between the spatial remote context is established to enhance the receptive field. GPN is used to enhance the information exchange between channels and guide the network to select appropriate multiscale features and spatial context semantic features. AFN is used to adaptively measure the importance of the above different features, and obtain accurate and effective feature representations from them. In addition, this article proposes a novel region-adaptive loss function, which optimizes the regions with large recognition errors in the image, and alleviates the inconsistency between the training target and the evaluation metric. In order to evaluate the performance of the proposed method, extensive experiments were carried out on challenging benchmarks including ShanghaiTech Part A and Part B, UCF-CC-50, UCF-QNRF, and JHU-CROWD ++ . Experimental results show that the proposed method has good performance on all four datasets. Especially on ShanghaiTech Part A and Part B, CUCF-QNRF, and JHU-CROWD ++ datasets, compared with the state-of-the-art methods, our proposed method achieves superior recognition performance and better robustness.

SAVING LIVES FROM ABOVE: PERSON DETECTION IN DISASTER RESPONSE USING DEEP NEURAL NETWORKS

Abstract. This paper focuses on person detection in aerial and drone imagery, which is crucial for various operations such as situational awareness, search and rescue, and safe delivery of supplies. We aim to improve disaster response efforts by enhancing the speed, safety, and effectiveness of the process. Therefore, we introduce a new person detection dataset comprising 311 annotated aerial and drone images, acquired from helicopters and drones in different scenes, including urban and rural areas, and for different scenarios, such as estimation of damage in disaster-affected zones, and search and rescue operations in different countries. The amount of data considered and level of detail of the annotations resulted in a total of 10,050 annotated persons. To detect people in aerial and drone images, we propose a multi-stage training procedure to improve YOLOv3’s ability. The proposed procedure aims at addressing challenges such as variations in scenes, scenarios, people poses, as well as image scales and viewing angles. To evaluate the effectiveness of our proposed training procedure, we split our dataset into a training and a test set. The latter includes images acquired during real search and rescue exercises and operations, and is therefore representative for the challenges encountered during operational missions and suitable for an accurate assessment of the proposed models. Experimental results demonstrate the effectiveness of our proposed training procedure, as the model’s average precision exhibits a relevant increase with respect to the baseline value.

CONSTRUCTION OF A DUAL-TASK MODEL FOR INDOOR SCENE RECOGNITION AND SEMANTIC SEGMENTATION BASED ON POINT CLOUDS

Abstract. Indoor scene recognition remains a challenging problem in the fields of artificial intelligence and computer vision due to the complexity, similarity, and spatial variability of indoor scenes. The existing research is mainly based on 2D data, which lacks 3D information about the scene and cannot accurately identify scenes with a high frequency of changes in lighting, shading, layout, etc. Moreover, the existing research usually focuses on the global features of the scene, which cannot represent indoor scenes with cluttered objects and complex spatial layouts. To solve the above problems, this paper proposes a dual-task model for indoor scene recognition and semantic segmentation based on point cloud data. The model expands the data loading method by giving the dataset loader the ability to return multi-dimensional labels and then realizes the dual-task model of scene recognition and semantic segmentation by fine-tuning PointNet++, setting task state control parameters, and adding a common feature layer. Finally, in order to solve the problem that the similarity of indoor scenes leads to the wrong scene recognition results, the rules of scenes and elements are constructed to correct the scene recognition results. The experimental results showed that with the assistance of scene-element rules, the overall accuracy of scene recognition with the proposed method in this paper is 82.4%, and the overall accuracy of semantic segmentation is 98.9%, which is better than the comparison model in this paper and provides a new method for cognition of indoor scenes based on 3D point clouds.

Direction-of-arrival and power spectral density estimation using a single directional microphone and group-sparse optimization

In this paper, two approaches are proposed for estimating the direction of arrival (DOA) and power spectral density (PSD) of stationary point sources by using a single, rotating, directional microphone. These approaches are based on a method previously presented by the authors, in which point source DOAs were estimated by using a broadband signal model and solving a group-sparse optimization problem, where the number of observations made by the rotating directional microphone can be lower than the number of candidate DOAs in an angular grid. The DOA estimation is followed by the estimation of the sources’ PSDs through the solution of an overdetermined least squares problem. The first approach proposed in this paper includes the use of an additional nonnegativity constraint on the residual noise term when solving the group-sparse optimization problem and is referred to as the Group Lasso Least Squares (GL-LS) approach. The second proposed approach, in addition to the new nonnegativity constraint, employs a narrowband signal model when building the linear system of equations used for formulating the group-sparse optimization problem, where the DOAs and PSDs can be jointly estimated by iterative, group-wise reweighting. This is referred to as the Group-Lasso with l_1-reweighting (GL-L1) approach. Both proposed approaches are implemented using the alternating direction method of multipliers (ADMM), and their performance is evaluated through simulations in which different setup conditions are considered, ranging from different types of model mismatch to variations in the acoustic scene and microphone directivity pattern. The results obtained show that in a scenario involving a microphone response mismatch between observed data and the signal model used, having the additional nonnegativity constraint on the residual noise can improve the DOA estimation for the case of GL-LS and the PSD estimation for the case of GL-L1. Moreover, the GL-L1 approach can present an advantage over GL-LS in terms of DOA estimation performance in scenarios with low SNR or where multiple sources are closely located to each other. Finally, it is shown that having the least squares PSD re-estimation step is beneficial in most scenarios, such that GL-LS outperformed GL-L1 in terms of PSD estimation errors.

Underwater image enhancement utilizing adaptive color correction and model conversion for dehazing

Underwater imaging is typically affected by absorption and scattering, which can lead to color distortion and image blurring. However, no universal color constancy algorithm for correcting underwater images exists due to the wide variation in underwater scenes. Furthermore, methods based on learning to improve underwater image quality often suffer from a lack of ground truth, which can compromise the validity and authenticity of restored images. In this study, we propose an adaptive underwater image enhancement technique that corrects image color and reduces haze using a valid dataset with ground truth. Specifically, we first use hue channel statistics from underwater images to build a dataset of color-corrected images from different underwater scenes. We use this dataset to train an Unet-like network for adaptive color correction. After color correction, underwater images often exhibit characteristics similar to hazy terrestrial images. We refer to this phenomenon as “model conversion”. The hazy terrestrial images have corresponding ground truth. Therefore, we train a Transformer-like network with hazy terrestrial image datasets to remove haze in underwater images. Our method is more robust for different underwater scenes and elegantly solves the problem of lacking ground truth. A series of experiments demonstrate that our method achieves superior performance compared to state-of-the-art methods in terms of both visual quality and quantitative metrics.

EFECL: Feature encoding enhancement with contrastive learning for indoor 3D object detection

Good proposal initials are critical for 3D object detection applications. However, due to the significant geometry variation of indoor scenes, incomplete and noisy proposals are inevitable in most cases. Mining feature information among these “bad” proposals may mislead the detection. Contrastive learning provides a feasible way for representing proposals, which can align complete and incomplete/noisy proposals in feature space. The aligned feature space can help us build robust 3D representation even if bad proposals are given. Therefore, we devise a new contrast learning framework for indoor 3D object detection, called EFECL, that learns robust 3D representations by contrastive learning of proposals on two different levels. Specifically, we optimize both instance-level and category-level contrasts to align features by capturing instance-specific characteristics and semantic-aware common patterns. Furthermore, we propose an enhanced feature aggregation module to extract more general and informative features for contrastive learning. Evaluations on ScanNet V2 and SUN RGB-D benchmarks demonstrate the generalizability and effectiveness of our method, and our method can achieve 12.3% and 7.3% improvements on both datasets over the benchmark alternatives. The code and models are publicly available at https://github.com/YaraDuan/EFECL.

Hierarchical attention aggregation with multi-resolution feature learning for GAN-based underwater image enhancement

In recent years, underwater image enhancement and restoration technologies have become increasingly important in order to optimize the efficiency of maritime operations and promote the automatic machine learning of the maritime industry. A new hierarchical attention aggregation with multi-resolution feature learning for GAN-based underwater image enhancement is proposed to address the problems of color bias, underexposure, and blurring in underwater images. The proposed method consists of a generator and a discriminator. Specifically, the generator includes an encoder, a bottleneck layer, and a decoder. Generator introduces inter-block serial connections for better adaptation to complex image scenes and task requirements, and parallel connections to extract multi-level features and enhance the expressive capacity of the network. To extract semantic and contextual information, hierarchical attention dense aggregation is designed in the encoder, which includes multi-scale feature hierarchy and dense feature hierarchy. Additionally, a multi-scale spatial attention mechanism is designed in the bottleneck layer to handle variations in underwater image scenes. In the decoder, the feature channel layer is emphasized, and a multi-channel attention mechanism is proposed to restore the multi-resolution channel features to a three-channel enhanced image. Furthermore, the angular loss function is introduced as additional supervision, which improves the similarity between the generated and original images, increases image clarity, and reduces color bias. Meanwhile, we employ the patch discriminator to enhance machine vision. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art methods.

Reduced spatiotemporal bandwidth of motion stabilized event-based imaging sensors: experimental demonstration.

This work investigates event-based sensor (EBS) imaging system's read-out bandwidth performance under linear motion, with and without hardware stabilization techniques. We implement three image stabilization methods using hardware rotation to cancel the sensor platform's linear motion and recapture lost EBS performance. We successfully demonstrated the methods, showing a bandwidth reduction of over an order of magnitude in two scenes, 10 scene variations, and five EBS velocities. This work demonstrates the benefits of stabilization with EBS to reduce bandwidth requirements versus unstabilized EBS systems.

Dilated Convolution-based Feature Refinement Network for Crowd Localization

As an emerging computer vision task, crowd localization has received increasing attention due to its ability to produce more accurate spatially predictions. However, continuous scale variations in complex crowd scenes lead to tiny individuals at the edges, so that existing methods cannot achieve precise crowd localization. Aiming at alleviating the above problems, we propose a novel Dilated Convolution-based Feature Refinement Network (DFRNet) to enhance the representation learning capability. Specifically, the DFRNet is built with three branches that can capture the information of each individual in crowd scenes more precisely. More specifically, we introduce a Feature Perception Module to model long-range contextual information at different scales by adopting multiple dilated convolutions, thus providing sufficient feature information to perceive tiny individuals at the edge of images. Afterwards, a Feature Refinement Module is deployed at multiple stages of the three branches to facilitate the mutual refinement of feature information at different scales, thus further improving the expression capability of multi-scale contextual information. By incorporating the above modules, DFRNet can locate individuals in complex scenes more precisely. Extensive experiments on multiple datasets demonstrate that the proposed method has more advanced performance compared to existing methods and can be more accurately adapted to complex crowd scenes.