Scene Classification Algorithms Research Articles

Retracted: Simulation of Tennis Match Scene Classification Algorithm Based on Adaptive Gaussian Mixture Model Parameter Estimation

Retracted: Simulation of Tennis Match Scene Classification Algorithm Based on Adaptive Gaussian Mixture Model Parameter Estimation

Collaborative Self-Supervised Transductive Few-Shot Learning for Remote Sensing Scene Classification

With the advent of deep learning and the accessibility of massive data, scene classification algorithms based on deep learning have been extensively researched and have achieved exciting developments. However, the success of deep models often relies on a large amount of annotated remote sensing data. Additionally, deep models are typically trained and tested on the same set of classes, leading to compromised generalization performance when encountering new classes. This is where few-shot learning aims to enable models to quickly generalize to new classes with only a few reference samples. In this paper, we propose a novel collaborative self-supervised transductive few-shot learning (CS2TFSL) algorithm for remote sensing scene classification. In our approach, we construct two distinct self-supervised auxiliary tasks to jointly train the feature extractor, aiming to obtain a powerful representation. Subsequently, the feature extractor’s parameters are frozen, requiring no further training, and transferred to the inference stage. During testing, we employ transductive inference to enhance the associative information between the support and query sets by leveraging additional sample information in the data. Extensive comparisons with state-of-the-art few-shot scene classification algorithms on the WHU-RS19 and NWPU-RESISC45 datasets demonstrate the effectiveness of the proposed CS2TFSL. More specifically, CS2TFSL ranks first in the settings of five-way one-shot and five-way five-shot. Additionally, detailed ablation experiments are conducted to analyze the CS2TFSL. The experimental results reveal significant and promising performance improvements in few-shot scene classification through the combination of self-supervised learning and direct transductive inference.

Region-Based Sea Ice Mapping Using Compact Polarimetric Synthetic Aperture Radar Imagery with Learned Features and Contextual Information

Operational sea ice maps are usually generated manually using dual-polarization (DP) synthetic aperture radar (SAR) satellite imagery, but there is strong interest in automating this process. Recently launched satellites offer compact polarimetry (CP) imagery that provides more comprehensive polarimetric information compared to DP, which compels the use of CP for automated classification of SAR sea ice imagery. Existing sea ice scene classification algorithms using CP imagery rely on handcrafted features, while neural networks offer the potential of features that are more discriminating. We have developed a new and effective sea ice classification algorithm that leverages the nature of CP data. First, a residual-based convolutional neural network (ResCNN) is implemented to classify each pixel. In parallel, an unsupervised segmentation is performed to generate regions based on CP statistical properties. Regions are assigned a single class label by majority voting using the ResCNN output. For testing, quad-polarimetric (QP) SAR sea ice scenes from the RADARSAT Constellation Mission (RCM) are used, and QP, DP, CP, and reconstructed QP modes are compared for classification accuracy, while also comparing them to other classification approaches. Using CP achieves an overall accuracy of 96.86%, which is comparable to QP (97.16%), and higher than reconstructed QP and DP data by about 2% and 10%, respectively. The implemented algorithm using CP imagery provides an improved option for automated sea ice mapping.

Multi-Scale Audio Spectrogram Transformer for Classroom Teaching Interaction Recognition

Classroom interactivity is one of the important metrics for assessing classrooms, and identifying classroom interactivity through classroom image data is limited by the interference of complex teaching scenarios. However, audio data within the classroom are characterized by significant student–teacher interaction. This study proposes a multi-scale audio spectrogram transformer (MAST) speech scene classification algorithm and constructs a classroom interactive audio dataset to achieve interactive teacher–student recognition in the classroom teaching process. First, the original speech signal is sampled and pre-processed to generate a multi-channel spectrogram, which enhances the representation of features compared with single-channel features; Second, in order to efficiently capture the long-range global context of the audio spectrogram, the audio features are globally modeled by the multi-head self-attention mechanism of MAST, and the feature resolution is reduced during feature extraction to continuously enrich the layer-level features while reducing the model complexity; Finally, a further combination with a time-frequency enrichment module maps the final output to a class feature map, enabling accurate audio category recognition. The experimental comparison of MAST is carried out on the public environment audio dataset and the self-built classroom audio interaction datasets. Compared with the previous state-of-the-art methods on public datasets AudioSet and ESC-50, its accuracy has been improved by 3% and 5%, respectively, and the accuracy of the self-built classroom audio interaction dataset has reached 92.1%. These results demonstrate the effectiveness of MAST in the field of general audio classification and the smart classroom domain.

Artificial intelligence-based collaborative acoustic scene and event classification to support urban soundscape analysis and classification

A human listener embedded in a sonic environment will rely on meaning given to sound events as well as on general acoustic features to analyse and appraise its soundscape. However, currently used measurable indicators for soundscape mainly focus on the latter and meaning is only included indirectly. Yet, today's artificial intelligence (AI) techniques allow to recognise a variety of sounds and thus assign meaning to them. Hence, we propose to combine a model for acoustic event classification trained on the large-scale environmental sound database AudioSet, with a scene classification algorithm that couples direct identification of acoustic features with these recognised sound for scene recognition. The combined model is trained on TUT2018, a database containing ten everyday scenes. Applying the resulting AI-model to the soundscapes of the world database without further training shows that the classification that is obtained correlates to perceived calmness and liveliness evaluated by a test panel. It also allows to unravel why an acoustic environment sounds like a lively square or a calm park by analysing the type of sounds and their occurrence pattern over time. Moreover, disturbance of the acoustic environment that is expected based on visual clues, by e.g. traffic can easily be recognised.

BiShuffleNeXt: A lightweight bi-path network for remote sensing scene classification

To address the problem of excessive computation in the remote sensing scene classification (RSSC) algorithms, this paper proposes an efficient RSSC model, which is named BiShuffleNeXt. Firstly, we use the sandglass bottleneck to design the ShuffleNeXt as a context path for deep semantic information extraction. Secondly, the spatial path consists of three sandglass bottlenecks, each with a stride of 2, to retain more spatial information and obtain a high-resolution feature map. Finally, the feature combination module is designed to perform the fusion of bi-path features. Experimental results show that the proposed model achieves a classification accuracy of 95.93% on the NWPU-45 dataset, which is a 3.6% improvement over using the context path alone. The proposed method is tested on three datasets and outperforms ShuffleNet v2, MobileNetv2, Mobile-Former and ResNet-50 respectively. The floating point operations (FLOPs) of BiShuffleNeXt are only 298.25M, which greatly reduces the calculation amount while ensuring classification accuracy.

Automatic User Preferences Selection of Smart Hearing Aid Using BioAid.

Noisy environments, changes and variations in the volume of speech, and non-face-to-face conversations impair the user experience with hearing aids. Generally, a hearing aid amplifies sounds so that a hearing-impaired person can listen, converse, and actively engage in daily activities. Presently, there are some sophisticated hearing aid algorithms available that operate on numerous frequency bands to not only amplify but also provide tuning and noise filtering to minimize background distractions. One of those is the BioAid assistive hearing system, which is an open-source, freely available downloadable app with twenty-four tuning settings. Critically, with this device, a person suffering with hearing loss must manually alter the settings/tuning of their hearing device when their surroundings and scene changes in order to attain a comfortable level of hearing. However, this manual switching among multiple tuning settings is inconvenient and cumbersome since the user is forced to switch to the state that best matches the scene every time the auditory environment changes. The goal of this study is to eliminate this manual switching and automate the BioAid with a scene classification algorithm so that the system automatically identifies the user-selected preferences based on adequate training. The aim of acoustic scene classification is to recognize the audio signature of one of the predefined scene classes that best represent the environment in which it was recorded. BioAid, an open-source biological inspired hearing aid algorithm, is used after conversion to Python. The proposed method consists of two main parts: classification of auditory scenes and selection of hearing aid tuning settings based on user experiences. The DCASE2017 dataset is utilized for scene classification. Among the many classifiers that were trained and tested, random forests have the highest accuracy of 99.7%. In the second part, clean speech audios from the LJ speech dataset are combined with scenes, and the user is asked to listen to the resulting audios and adjust the presets and subsets. A CSV file stores the selection of presets and subsets at which the user can hear clearly against the scenes. Various classifiers are trained on the dataset of user preferences. After training, clean speech audio was convolved with the scene and fed as input to the scene classifier that predicts the scene. The predicted scene was then fed as input to the preset classifier that predicts the user’s choice for preset and subset. The BioAid is automatically tuned to the predicted selection. The accuracy of random forest in the prediction of presets and subsets was 100%. This proposed approach has great potential to eliminate the tedious manual switching of hearing assistive device parameters by allowing hearing-impaired individuals to actively participate in daily life by automatically adjusting hearing aid settings based on the acoustic scene.

CAW: A Remote-Sensing Scene Classification Network Aided by Local Window Attention.

Remote-sensing image scene data contain a large number of scene images with different scales. Traditional scene classification algorithms based on convolutional neural networks are difficult to extract complex spatial distribution and texture information in images, resulting in poor classification results. In response to the above problems, we introduce the vision transformer network structure with strong global modeling ability into the remote-sensing image scene classification task. In this paper, the parallel network structure of the local-window self-attention mechanism and the equivalent large convolution kernel is used to realize the spatial-channel modeling of the network so that the network has better local and global feature extraction performance. Experiments on the RSSCN7 dataset and the WHU-RS19 dataset show that the proposed network can improve the accuracy of scene classification. At the same time, the effectiveness of the network structure in remote-sensing image classification tasks is verified through ablation experiments, confusion matrix, and heat map results comparison.

A Lightweight Self-Supervised Representation Learning Algorithm for Scene Classification in Spaceborne SAR and Optical Images

Despite the increasing amount of spaceborne synthetic aperture radar (SAR) images and optical images, only a few annotated data can be used directly for scene classification tasks based on convolution neural networks (CNNs). For this situation, self-supervised learning methods can improve scene classification accuracy through learning representations from extensive unlabeled data. However, existing self-supervised scene classification algorithms are hard to deploy on satellites, due to the high computation consumption. To address this challenge, we propose a simple, yet effective, self-supervised representation learning (Lite-SRL) algorithm for the scene classification task. First, we design a lightweight contrastive learning structure for Lite-SRL, we apply a stochastic augmentation strategy to obtain augmented views from unlabeled spaceborne images, and Lite-SRL maximizes the similarity of augmented views to learn valuable representations. Then, we adopt the stop-gradient operation to make Lite-SRL’s training process not rely on large queues or negative samples, which can reduce the computation consumption. Furthermore, in order to deploy Lite-SRL on low-power on-board computing platforms, we propose a distributed hybrid parallelism (DHP) framework and a computation workload balancing (CWB) module for Lite-SRL. Experiments on representative datasets including OpenSARUrban, WHU-SAR6, NWPU-Resisc45, and AID dataset demonstrate that Lite-SRL can improve the scene classification accuracy under limited annotated data, and it is generalizable to both SAR and optical images. Meanwhile, compared with six state-of-the-art self-supervised algorithms, Lite-SRL has clear advantages in overall accuracy, number of parameters, memory consumption, and training latency. Eventually, to evaluate the proposed work’s on-board operational capability, we transplant Lite-SRL to the low-power computing platform NVIDIA Jetson TX2.

Beamspace Scene Classification Algorithm for Low-Angle Estimation in MIMO Radar

This paper discusses the issue of low-angle estimation in multiple-input, multiple-output (MIMO) radar. Due to the low amount of data transmission, storage, and computation required for beamspace super-resolution algorithms in low-angle estimation, the method has gained considerable interest in recent years. This paper develops a beamspace scene classification (BSC) algorithm to enhance the performance of low-angle estimation in MIMO radar. The proposed BSC algorithm solves an initial angle estimate and the multipath coefficient estimate using the 3D beamspace data, and further constructs the beamspace data required for transmitting and receiving sides to reduce the data dimensions. It is additionally used to provide three estimation schemes with closed-form solutions for three multipath scenes (obtaining two estimations for transmitting and receiving sides). Finally, it fuses the estimates of the two sides by the minimum variance criterion. As a result, the proposed method achieves high estimation accuracy while requiring few processing resources. Moreover, the computational complexity of the proposed algorithm is examined in this study, with the results demonstrating that the proposed method is superior for engineering applications.

Ice Detection with Sentinel-1 SAR Backscatter Threshold in Long Sections of Temperate Climate Rivers

Climate change leads to more variable meteorological conditions. In many Northern Hemisphere temperate regions, cold seasons have become more variable and unpredictable, necessitating frequent river ice observations over long sections of rivers. Satellite SAR (Synthetic Aperture Radar)-based river ice detection models have been successfully applied and tested, but different hydrological, morphological and climatological conditions can affect their skill. In this study, we developed and tested Sentinel-1 SAR-based ice detection models in 525 km sections of the Nemunas and Neris Rivers. We analyzed three binary classification models based on VV, VH backscatter and logistic regression. The model sensitivity and specificity were used to determine the optimal threshold between ice and water classes. We used in situ observations and Sentinel-2 Sen2Cor ice mask to validate models in different ice conditions. In most cases, SAR-based ice detection models outperformed Sen2Cor classification because Sen2Cor misclassified pixels as ice in areas with translucent clouds, undetected by the scene classification algorithm, and misclassified pixels as water in cloud or river valley shadow. SAR models were less accurate in river sections where river flow and ice formation conditions were affected by large valley-dammed reservoirs. Sen2Cor and SAR models accurately detected border and consolidated ice but were less accurate in moving ice conditions. The skill of models depended on how dense the moving ice was. With a lowered classification threshold and increased model sensitivity, SAR models detected sparse frazil ice. In most cases, the VV polarization-based model was more accurate than the VH polarization-based model. The results of logistic and VV models were highly correlated, and the use of VV was more constructive due to its simpler algorithm.

Structural Attention Enhanced Continual Meta-Learning for Graph Edge Labeling Based Few-Shot Remote Sensing Scene Classification

Scene classification is one of the fundamental techniques shared by many basic remote sensing tasks with a wide range of applications. As the demands of catering with situations under high variance in the data urgent conditions are rising, a research topic called few-shot scene classification is receiving more interest with a focus on building classification model from few training samples. Currently, methods using the meta-learning principle or graphical models are achieving state-of-art performances. However, there are still significant gaps in between the few-shot methods and the traditionally trained ones, as there are implicit data isolations in standard meta-learning procedure and less-flexibility in the static graph neural network modeling technique, which largely limit the data-to-knowledge transition efficiency. To address these issues, this paper proposed an novel few-shot scene classification algorithm based on a different meta-learning principle called continual meta-learning, which enhances the inter-task correlation by fusing more historical prior knowledge from a sequence of tasks within sections of meta-training or meta-testing periods. Moreover, as to increase the discriminative power between classes, a graph transformer is introduced to produce the structural attention, which can optimize the distribution of sample features in the embedded space and promotes the overall classification capability of the model. The advantages of our proposed algorithm are verified by comparing with nine state-of-art meta-learning based on few-shot scene classification on three popular datasets, where a minimum of a 9% increase in accuracy can be observed. Furthermore, the efficiency of the newly added modular modifications have also be verified by comparing to the continual meta-learning baseline.

Machine Learning with Remote Sensing Image Data Sets

Computer vision, as a part of machine learning, gains significant attention from researches nowadays. Aerial scene classification is a prominent chapter of computer vision with a vast application: military, surveillance and security, environment monitoring, detection of geospatial objects, etc. There are several publicly available remote sensing image datasets, which enable the deployment of various aerial scene classification algorithms. In our article, we use transfer learning from pre-trained deep Convolutional Neural Networks (CNN) within remote sensing image classification. Neural networks utilized in our research are high-dimensional previously trained CNN on ImageNet dataset. Transfer learning can be performed through feature extraction or fine-tuning. We proposed a two-stream feature extraction method and afterward image classification through a handcrafted classifier. Fine-tuning was performed with adaptive learning rates and a regularization method label smoothing. The proposed transfer learning techniques were validated on two remote sensing image datasets: WHU RS datasets and AID dataset. Our proposed method obtained competitive results compared to state-of-the-art methods.

Simulation of Tennis Match Scene Classification Algorithm Based on Adaptive Gaussian Mixture Model Parameter Estimation

This paper presents an in-depth analysis of tennis match scene classification using an adaptive Gaussian mixture model parameter estimation simulation algorithm. We divided the main components of semantic analysis into type of motion, distance of motion, speed of motion, and landing area of the tennis ball. Firstly, for the problem that both people and tennis balls in the video frames of tennis matches from the surveillance viewpoint are very small, we propose an adaptive Gaussian mixture model parameter estimation algorithm, which has good accuracy and speed on small targets. Secondly, in this paper, we design a sports player tracking algorithm based on role division and continuously lock the target player to be tracked and output the player region. At the same time, based on the displacement information of the key points of the player’s body and the system running time, the distance and speed of the player’s movement are obtained. Then, for the problem that tennis balls are small and difficult to capture in high-speed motion, this paper designs a prior knowledge-based algorithm for predicting tennis ball motion and landing area to derive the landing area of tennis balls. Finally, this paper implements a prototype system for semantic analysis of real-time video of tennis matches and tests and analyzes the performance indexes of the system, and the results show that the system has good performance in real-time, accuracy, and stability.

Channel Sounding and Scene Classification of Indoor 6G Millimeter Wave Channel Based on Machine Learning

Millimeter wave, especially the high frequency millimeter wave near 100 GHz, is one of the key spectrum resources for the sixth generation (6G) mobile communication, which can be used for precise positioning, imaging and large capacity data transmission. Therefore, high frequency millimeter wave channel sounding is the first step to better understand 6G signal propagation. Because indoor wireless deployment is critical to 6G and different scenes classification can make future radio network optimization easy, we built a 6G indoor millimeter wave channel sounding system using just commercial instruments based on time-domain correlation method. Taking transmission and reception of a typical 93 GHz millimeter wave signal in the W-band as an example, four indoor millimeter wave communication scenes were modeled. Furthermore, we proposed a data-driven supervised machine learning method to extract fingerprint features from different scenes. Then we trained the scene classification model based on these features. Baseband data from receiver was transformed to channel Power Delay Profile (PDP), and then six fingerprint features were extracted for each scene. The decision tree, Support Vector Machine (SVM) and the optimal bagging channel scene classification algorithms were used to train machine learning model, with test accuracies of 94.3%, 86.4% and 96.5% respectively. The results show that the channel fingerprint classification model trained by machine learning method is effective. This method can be used in 6G channel sounding and scene classification to THz in the future.

L2-norm prototypical networks for tackling the data shift problem in scene classification

ABSTRACT Currently, most scene classification algorithms are trained and evaluated based on a single dataset. However, practical applications are not usually restricted to specific satellite platforms or datasets. Researchers generally train a model on one dataset and require the model to perform tasks with another dataset, leading to a data shift problem and performance decrease in practical applications. To address this problem, this study presents a metric-based few-shot classification method with -norm prototypical networks. Specifically, a carefully designed -norm layer was introduced into prototypical networks. The proposed -norm layer applies -norm operations to prototypes and query features to mitigate the length fluctuations caused by the data shift problem. With this layer, the -norm prototypical networks maintain the ability to identify novel classes and limit the effects of data discrepancies. The proposed -norm layer improves the classification accuracy by 0.42% to 2.41% on various public datasets. Moreover, -norm prototypical networks outperform other methods by 0.02% to 34.38%. Comprehensive experiments consistently demonstrate the advantages of the proposed method in tackling the data shift problem.

Interleaved Acoustic Environments: Impact of an Auditory Scene Classification Procedure on Speech Perception in Cochlear Implant Users.

Clinical speech perception tests with simple presentation conditions often overestimate the impact of signal preprocessing on speech perception in complex listening environments. A new procedure was developed to assess speech perception in interleaved acoustic environments of different complexity that allows investigation of the impact of an automatic scene classification (ASC) algorithm on speech perception. The procedure was applied in cohorts of normal hearing (NH) controls and uni- and bilateral cochlear implant (CI) users. Speech reception thresholds (SRTs) were measured by means of a matrix sentence test in five acoustic environments that included different noise conditions (amplitude modulated and continuous), two spatial configurations, and reverberation. The acoustic environments were encapsulated in a randomized, mixed order single experimental run. Acoustic room simulation was played back with a loudspeaker auralization setup with 128 loudspeakers. 18 NH, 16 unilateral, and 16 bilateral CI users participated. SRTs were evaluated for each individual acoustic environment and as mean-SRT. Mean-SRTs improved by 2.4 dB signal-to-noise ratio for unilateral and 1.3 dB signal-to-noise ratio for bilateral CI users with activated ASC. Without ASC, the mean-SRT of bilateral CI users was 3.7 dB better than the SRT of unilateral CI users. The mean-SRT indicated significant differences, with NH group performing best and unilateral CI users performing worse with a difference of up to 13 dB compared to NH. The proposed speech test procedure successfully demonstrated that speech perception and benefit with ASC depend on the acoustic environment.

Place Classification Algorithm Based on Semantic Segmented Objects

Scene or place classification is one of the important problems in image and video search and recommendation systems. Humans can understand the scene they are located, but it is difficult for machines to do it. Considering a scene image which has several objects, humans recognize the scene based on these objects, especially background objects. According to this observation, we propose an efficient scene classification algorithm for three different classes by detecting objects in the scene. We use pre-trained semantic segmentation model to extract objects from an image. After that, we construct a weight matrix to determine a scene class better. Finally, we classify an image into one of three scene classes (i.e., indoor, nature, city) by using the designed weighting matrix. The performance of our scheme outperforms several classification methods using convolutional neural networks (CNNs), such as VGG, Inception, ResNet, ResNeXt, Wide-ResNet, DenseNet, and MnasNet. The proposed model achieves 90.8% of verification accuracy and improves over 2.8% of the accuracy when comparing to the existing CNN-based methods.

Cochlear implant datalogging accurately characterizes children’s ‘auditory scenes’

Objectives: This study sought to determine if children’s auditory environments are accurately captured by the automatic scene classification embedded in cochlear implant (CI) processors and to quantify the amount of electronic device use in these environments. Methods: Seven children with CIs, 36.71 (SD = 11.94) months old, participated in this study. Three of the children were male and four were female. Eleven datalogs, containing outcomes from Cochlear’s™ Nucleus® 6 (Cochlear Corporation, Australia) CI scene classification algorithm, and seven day-long audio recordings collected with a Language ENvironment Analysis (LENA; LENA Research Foundation, USA) recorder were obtained for analysis. Results: Results from the scene classification algorithm were strongly correlated with categories determined through human coding (ICC = .86, CI = [−0.2, 1], F(5,5.1) = 5.9, P = 0.04) but some differences emerged. Scene classification identified more ‘Quiet’ (t(8.2) = 4.1, P = 0.003) than human coders, while humans identified more ‘Speech’ (t(10.6) = −2.4, P = 0.04). On average, 8% (SD = 5.8) of the children’s day was spent in electronic sound, which was primarily produced by mobile devices (39.7%). Discussion: While CI scene classification software reflects children’s natural auditory environments, it is important to consider how different scenes are defined when interpreting results. An electronic sounds category should be considered given how often children are exposed to such sounds.

AutoRSISC: Automatic design of neural architecture for remote sensing image scene classification

• A remote sensing image scene classification algorithm base on neural architecture search (NAS) is proposed. • A search strategy is proposed to search for the most appropriate neural architecture in the pre-defined search space. • A performance estimation strategy to evaluate the performance of search strategy. • The proposed method saves time and manpower while achieves better accuracy compared with the manually design method. Remote sensing image scene classification (RSISC) is one of the bases of visual question answering for remote sensing data (RSVQA). In the conventional deep learning based solutions of RSISC, domain experts have to manually design the neural architecture, which is costly in both time and manpower. In this paper, AutoRSISC, a high resolution remote sensing image scene classification method based on neural architecture search (NAS) is proposed. AutoRSISC algorithm samples the neural architecture in a certain proportion to reduce the redundancy in the search space which is based on the continuous relaxation of the neural architecture representation. Edge normalization is introduced to make architecture search more efficient. The conventional deep learning approaches take days or more just to manually design the appropriate neural architecture according to the current data set. The experimental results show that our method saves time and manpower compared with the existing manually designed RSISC methods. Our AutoRSISC algorithm took 7.6 h from the automatic design of the most appropriate neural architecture to the end of all experiments according to the UCM data set, and the classification accuracy rate of the final test set reached 97.85%. And our AutoRSISC algorithm took 101.7 h from the automatic design of the most appropriate neural architecture to the end of all experiments according to the NWPU45 data set, and the classification accuracy rate of the final test set reached 94.66%.