Visual Feature Extraction Research Articles

Automated Video Events Detection and Classification using CNN-GRU Model

In the era of vast and continuous video content creation, manually identifying crucial events becomes a tedious and inefficient task. To address this challenge, we propose a CNN-GRU model that automatically detects and classifies significant events in videos. This model employs ResNet50 Convolutional Neural Networks (CNNs) to extract visual features from video frames, followed by Gated Recurrent Units (GRUs) for temporal modelling and event recognition. By leveraging the sequential data handling capabilities of GRUs, our model captures temporal patterns across frames. We evaluate the model's performance using accuracy and F1-score metrics on the VIRAT dataset, containing 1,555 events across 12 event classes. Our approach achieves promising results, with an event classification accuracy of 75.22%.

White shark optimizer with optimal deep learning based effective unmanned aerial vehicles communication and scene classification

Unmanned aerial vehicles (UAVs) become a promising enabler for the next generation of wireless networks with the tremendous growth in electronics and communications. The application of UAV communications comprises messages relying on coverage extension for transmission networks after disasters, Internet of Things (IoT) devices, and dispatching distress messages from the device positioned within the coverage hole to the emergency centre. But there are some problems in enhancing UAV clustering and scene classification using deep learning approaches for enhancing performance. This article presents a new White Shark Optimizer with Optimal Deep Learning based Effective Unmanned Aerial Vehicles Communication and Scene Classification (WSOODL-UAVCSC) technique. UAV clustering and scene categorization present many deep learning challenges in disaster management: scene understanding complexity, data variability and abundance, visual data feature extraction, nonlinear and high-dimensional data, adaptability and generalization, real-time decision making, UAV clustering optimization, sparse and incomplete data. the need to handle complex, high-dimensional data, adapt to changing environments, and make quick, correct decisions in critical situations drives deep learning in UAV clustering and scene categorization. The purpose of the WSOODL-UAVCSC technique is to cluster the UAVs for effective communication and scene classification. The WSO algorithm is utilized for the optimization of the UAV clustering process and enables to accomplish effective communication and interaction in the network. With dynamic adjustment of the clustering, the WSO algorithm improves the performance and robustness of the UAV system. For the scene classification process, the WSOODL-UAVCSC technique involves capsule network (CapsNet) feature extraction, marine predators algorithm (MPA) based hyperparameter tuning, and echo state network (ESN) classification. A wide-ranging simulation analysis was conducted to validate the enriched performance of the WSOODL-UAVCSC approach. Extensive result analysis pointed out the enhanced performance of the WSOODL-UAVCSC method over other existing techniques. The WSOODL-UAVCSC method achieved an accuracy of 99.12%, precision of 97.45%, recall of 98.90%, and F1-score of 98.10% when compared to other existing techniques.

MRAN: Multimodal relationship-aware attention network for fake news detection

Existing multimodal fake news detection methods face challenges in jointly capturing the intramodality and cross-modal correlation relationships between image regions and text fragments. Additionally, these methods lack comprehensive hierarchical semantics mining for text. These limitations result in ineffective utilization of multimodal information and impact detection performance. To address these issues, we propose a multimodal relationship-aware attention network (MRAN), which consists of three main steps. First, a multi-level encoding network is employed to extract hierarchical semantic feature representations of text, while the visual feature extractor VGG19 learns image feature representations. Second, the captured text and image representations are input into the relationship-aware attention network, which generates high-order fusion features by calculating the similarity between information segments within modalities and cross-modal similarity. Finally, the fusion features are passed through a fake news detector, which identifies fake news. Experimental results on three benchmark datasets demonstrate the effectiveness of MRAN, highlighting its strong detection performance.

A SUPERPOINT NEURAL NETWORK IMPLEMENTATION FOR ACCURATE FEATURE EXTRACTION IN UNSTRUCTURED ENVIRONMENTS

Abstract. Feature extraction plays a crucial role in visual localization, SLAM (Simultaneous Localization and Mapping) and autonomous navigation, by enabling the extraction and tracking of distinctive visual features for both mapping and localization tasks. However, most of the studies, investigate the efficiency and performance of the algorithms in urban, vegetated or indoor environments and not in unstructured environments which suffers by poor information in visual cues where a feature extraction algorithm or architecture could base on. In this study, an investigation of SuperPoint architecture’s efficiency in keypoint detection and description applied to unstructured and planetary scenes was conducted, producing three different models: (a) an original SuperPoint model trained from scratch, (b) an original fine-tuned SuperPoint model, (c) an optimized SuperPoint model, trained from scratch with the same parametarization as the corresponding original model. For the training process, a dataset of 48 000 images was utilized representing unstructured scenes from Earth, Moon and Mars while a benchmark dataset was used aiming to evaluate the model in illumination and viewpoint changes. The experimentation proved that the optimized SuperPoint model provides superior performance using repeatability and homography estimation metrics, compared with the original SuperPoint models, and handcrafted keypoint detectors and descriptors.

Real Time Accident Detection from Closed Circuit Television and Suggestion of Nearest Medical Amenities

The prevalence of automobile accidents as a major cause of violent deaths around the world has prompted researchers to develop an automated method for detecting them. The effectiveness of medical response to accident scenes and the chances of survival are influenced by the human element, underscoring the need for an automated system. With the widespread use of video surveillance and advanced traffic systems, researchers have proposed a model to automatically detect traffic accidents on video. The proposed approach assumes that visual elements occurring in a temporal sequence correspond to traffic accidents. The model architecture consists of two phases: visual feature extraction and temporal pattern detection. Convolution and recurrent layers are employed during training to learn visual and temporal features from scratch as well as from publicly available datasets. The proposed accident detection and alerting system using Convolution Neural Network models with Rectified Linear Unit and Softmax activation functions is an effective tool for detecting different types of accidents in real-time. The system of accident detection, integrated with the alerting mechanism for prompt medical assistance achieved high accuracy and recall rates.

Multi-modal Deep Learning Approach to Improve Sentence level Sinhala Sign Language Recognition

Sign language is used across the world for communication purposes within hearing-impaired communities. Hearing people are not well versed in sign language and most hearing-impaired are not good in general text, creating a communication barrier. Research on Sign Language Recognition (SLR) systems have shown admirable solutions for this issue. In Sri Lanka, machine learning along with neural networks has been the prominent domain of research in Sinhala SLR. All previous research is mainly focused on word-level SLR using hand gestures for translation. While this works for a certain vocabulary, there are many signs interpreted through other spatial cues like lip movements and facial expressions. Therefore, translation is limited and sometimes the interpretations can be misleading. In this research, we propose a multi-modal Deep Learning approach that can effectively recognize sentence-level sign gestures using hand and lip movements and translate to Sinhala text. The model consists of modules for visual feature extraction (ResNet), contextual relationship modeling (transformer encoder with multi-head attention), alignment (CTC) and decoding (Prefix beam search). A dataset consisting 22 of sentences used for evaluations was collected under controlled conditions for a specific day-to-day scenario (a conversation between a vendor and a customer in a shop). The proposed model achieves a best Word Error Rate (WER) of 12.70 on the testing split, improving over the single-stream model which shows a best WER of 17.41, suggesting a multi-modal approach improves overall SLR.

The Application of Multimodal Discourse Analysis in Urban Intercultural Communication

Abstract This paper firstly analyzes the basic composition of intercultural communication ability, discusses the specific function of cultural promotion of urban tourism, and gives the promotion effect of intercultural communication on urban tourism. Secondly, the connotation of multimodal discourse analysis is given, and the technical analysis of text, audio, and visual features is carried out for the analysis of urban tourism discourse. Then, the TF-IDF algorithm is used to realize the feature extraction of tourism culture text, the MFCC algorithm is used to extract the audio features of tourism culture, and the modal classification network is used to realize the recognition of the visual features of urban tourism culture video. Finally, to verify the effectiveness of the application of multimodal discourse analysis in urban cross-cultural communication, three aspects were tested and analyzed respectively. The results show that the F1 value of the TF-IDF algorithm is 0.912, which is 17.07% higher than that of the CTF-TF-IDF algorithm. When the amount of audio recognition is 5GB, the recognition time of the MFCC audio recognition method is 10.4 s. When the weight value of the visual feature extraction network is set to 1.0, the highest visual feature extraction error rate is only 3.96%. Using multimodal discourse analysis to carry out urban tourism discourse analysis can realize more comprehensive urban tourism feature extraction, help tourists strengthen their tourism perception, and then promote the enhancement of urban cross-cultural communication ability.

Language-Model-Based Paired Variational Autoencoders for Robotic Language Learning

Human infants learn language while interacting with their environment in which their caregivers may describe the objects and actions they perform. Similar to human infants, artificial agents can learn language while interacting with their environment. In this work, first, we present a neural model that bidirectionally binds robot actions and their language descriptions in a simple object manipulation scenario. Building on our previous Paired Variational Autoencoders (PVAE) model, we demonstrate the superiority of the variational autoencoder over standard autoencoders by experimenting with cubes of different colours, and by enabling the production of alternative vocabularies. Additional experiments show that the model's channel-separated visual feature extraction module can cope with objects of different shapes. Next, we introduce PVAE-BERT, which equips the model with a pretrained large-scale language model, i.e., Bidirectional Encoder Representations from Transformers (BERT), enabling the model to go beyond comprehending only the predefined descriptions that the network has been trained on; the recognition of action descriptions generalises to unconstrained natural language as the model becomes capable of understanding unlimited variations of the same descriptions. Our experiments suggest that using a pretrained language model as the language encoder allows our approach to scale up for real-world scenarios with instructions from human users.

A Deep-Learning-Based Multimodal Data Fusion Framework for Urban Region Function Recognition

Accurate and efficient classification maps of urban functional zones (UFZs) are crucial to urban planning, management, and decision making. Due to the complex socioeconomic UFZ properties, it is increasingly challenging to identify urban functional zones by using remote-sensing images (RSIs) alone. Point-of-interest (POI) data and remote-sensing image data play important roles in UFZ extraction. However, many existing methods only use a single type of data or simply combine the two, failing to take full advantage of the complementary advantages between them. Therefore, we designed a deep-learning framework that integrates the above two types of data to identify urban functional areas. In the first part of the complementary feature-learning and fusion module, we use a convolutional neural network (CNN) to extract visual features and social features. Specifically, we extract visual features from RSI data, while POI data are converted into a distance heatmap tensor that is input into the CNN with gated attention mechanisms to extract social features. Then, we use a feature fusion module (FFM) with adaptive weights to fuse the two types of features. The second part is the spatial-relationship-modeling module. We designed a new spatial-relationship-learning network based on a vision transformer model with long- and short-distance attention, which can simultaneously learn the global and local spatial relationships of the urban functional zones. Finally, a feature aggregation module (FGM) utilizes the two spatial relationships efficiently. The experimental results show that the proposed model can fully extract visual features, social features, and spatial relationship features from RSIs and POIs for more accurate UFZ recognition.

UiAnalyzer: Evaluating whether the UI of apps is at the risk of violating the design conventions in terms of function layout

There are many unofficial but widely accepted design conventions on the function layout of the user interface (UI) of apps, which include the position of functions and position relationship between functions. If an app fails to follow these design conventions, users may feel uncomfortable when using the app and this can negatively affect its market value. Unfortunately, no research is available to help app developers analyze whether the UI of their products is at the risk of violating the design conventions in terms of the function layout. To fill this gap, we propose a novel function layout analysis method called UiAnalyzer. This method compares the function layout of a UI with that of similar UIs to evaluate the acceptability of its function layout. Specifically, UiAnalyzer first generates semantic wireframes for the analyzed UI and similar UIs according to their function layout. Then, it extracts visual features from the semantic wireframes using a convolutional neural network and applies an anomaly detection algorithm to process the obtained visual features to determine whether the visual feature corresponding to the analyzed UI is anomalous. The experiment based on Google Play apps shows that UiAnalyzer is capable of evaluating the function layout of UIs of apps.

A hierarchical fine-grained classification approach for COVID-19 severity assessment based on CT images

The Coronavirus Disease 2019 (COVID-19) is still an ongoing health issue. Appropriate treatment is important for the recovery of COVID-19 patients. At present, numerous deep learning (DL) methods have been applied to the classification of COVID-19 based on computerized tomography (CT) images. However, most of these methods directly extract visual features from the entire CT image, leading to the inclusion of irrelevant background information and making it hard to extract critical features about lesions. Thus, we propose a hierarchical fine-grained classification approach for automatic severity assessment of COVID-19 patients. The CT images are analyzed from coarse to fine in two stages, to focus on the critical features gradually. In the first stage, we detect the lesion regions from the CT image utilizing deep learning based model. In the second stage, we extract critical features from lesion regions and make a severity grading by fine-grained classification model. The experimental results show that the proposed method achieves the classification results of 95.04%, 93.98%, 93.46% and 93.72% in the metrics of accuracy, precision, recall and F1-Score, respectively.

Artificial intelligence for automatic surgical phase recognition of laparoscopic gastrectomy in gastric cancer.

This study aimed to classify laparoscopic gastric cancer phases. We also aimed to develop a transformer-based artificial intelligence (AI) model for automatic surgical phase recognition and to evaluate the model's performance using laparoscopic gastric cancer surgical videos. One hundred patients who underwent laparoscopic surgery for gastric cancer were included in this study. All surgical videos were labeled and classified into eight phases (P0. Preparation. P1. Separate the greater gastric curvature. P2. Separate the distal stomach. P3. Separate lesser gastric curvature. P4. Dissect the superior margin of the pancreas. P5. Separation of the proximal stomach. P6. Digestive tract reconstruction. P7. End of operation). This study proposed an AI phase recognition model consisting of a convolutional neural network-based visual feature extractor and temporal relational transformer. A visual and temporal relationship network was proposed to automatically perform accurate surgical phase prediction. The average time for all surgical videos in the video set was 9114 ± 2571s. The longest phase was at P1 (3388s). The final research accuracy, F1, recall, and precision were 90.128, 87.04, 87.04, and 87.32%, respectively. The phase with the highest recognition accuracy was P1, and that with the lowest accuracy was P2. An AI model based on neural and transformer networks was developed in this study. This model can identify the phases of laparoscopic surgery for gastric cancer accurately. AI can be used as an analytical tool for gastric cancer surgical videos.

Multimodal fusion sensitive information classification based on mixed attention and CLIP model1

Social network attackers leverage images and text to disseminate sensitive information associated with pornography, politics, and terrorism,causing adverse effects on society.The current sensitive information classification model does not focus on feature fusion between images and text, greatly reducing recognition accuracy.To address this problem, we propose an attentive cross-modal fusion model (ACMF), which utilizes mixed attention mechanism and the Contrastive Language-Image Pre-training model.Specifically, we employ a deep neural network with a mixed attention mechanism as a visual feature extractor. This allows us to progressively extract features at different levels. We combine these visual features with those obtained from a text feature extractor and incorporate image-text frequency domain information at various levels to enable fine-grained modeling. Additionally, we introduce a cyclic attention mechanism and integrate the Contrastive Language-Image Pre-training model to establish stronger connections between modalities, thereby enhancing classification performance.Experimental evaluations conducted on sensitive information datasets collected demonstrate the superiority of our method over other baseline models. The model achieves an accuracy rate of 91.4% and an F1-score of 0.9145. These results validate the effectiveness of the mixed attention mechanism in enhancing the utilization of important features. Furthermore, the effective fusion of text and image features significantly improves the classification ability of the deep neural network.

Audiovisual speech recognition based on a deep convolutional neural network

Audiovisual speech recognition is an emerging research topic. Lipreading is the recognition of what someone is saying using visual information, primarily lip movements. In this study, we created a custom dataset for Indian English linguistics and categorized it into three main categories: (1) audio recognition, (2) visual feature extraction, and (3) combined audio and visual recognition. Audio features were extracted using the mel-frequency cepstral coefficient, and classification was performed using a one-dimension convolutional neural network. Visual feature extraction uses Dlib and then classifies visual speech using a long short-term memory type of recurrent neural networks. Finally, integration was performed using a deep convolutional network. The audio speech of Indian English was successfully recognized with accuracies of 93.67% and 91.53%, respectively, using testing data from 200 epochs. The training accuracy for visual speech recognition using the Indian English dataset was 77.48% and the test accuracy was 76.19% using 60 epochs. After integration, the accuracies of audiovisual speech recognition using the Indian English dataset for training and testing were 94.67% and 91.75%, respectively.

Facial expression recognition based on adaptation of the classifier to videos of the user

In this paper, an approach that can significantly increase the accuracy of facial emotion recognition by adapting the model to the emotions of a particular user (e.g., smartphone owner) is considered. At the first stage, a neural network model, which was previously trained to recognize facial expressions in static photos, is used to extract visual features of faces in each frame. Next, the face features of video frames are aggregated into a single descriptor for a short video fragment. After that a neural network classifier is trained. At the second stage, it is proposed that adaptation (fine-tuning) to this classifier should be performed using a small set of video data with the facial expressions of a particular user. After emotion classification, the user can adjust the predicted emotions to further improve the accuracy of a personal model. As part of an experimental study for the RAVDESS dataset, it has been shown that the approach with model adaptation to a specific user can significantly (up to 20 – 50 %) improve the accuracy of facial expression recognition in the video.

A Multi-Task Road Feature Extraction Network with Grouped Convolution and Attention Mechanisms.

To cope with the challenges of autonomous driving in complex road environments, the need for collaborative multi-tasking has been proposed. This research direction explores new solutions at the application level and has become a hot topic of great interest. In the field of natural language processing and recommendation algorithms, the use of multi-task learning networks has been proven to reduce time, computing power, and storage usage in various task coupling cases. Due to the characteristics of the multi-task learning network, it has also been applied to visual road feature extraction in recent years. This article proposes a multi-task road feature extraction network that combines group convolution with transformer and squeeze excitation attention mechanisms. The network can simultaneously perform drivable area segmentation, lane line segmentation, and traffic object detection tasks. The experimental results of the BDD-100K dataset show that the proposed method performs well for different tasks and has a higher accuracy than similar algorithms. The proposed method provides new ideas and methods for the autonomous road perception of vehicles and the generation of highly accurate maps in visual-based autonomous driving processes.

Deep Seasonal Network for Remote Sensing Imagery Classification of Multi-Temporal Sentinel-2 Data

As a medium-resolution multi-temporal data source, Sentinel-2 data has the potential to match the performance of using very-high-resolution (VHR) images in deep learning applications. To fully leverage the multi-temporal nature of Sentinel-2 data, we introduce the Deep Seasonal Network (DeepSN). This composite architecture combines a pre-trained deep convolutional neural network (DCNN) for visual feature extraction with a long short-term memory (LSTM) model to capture temporal information and make classification predictions. We evaluate the effectiveness of DeepSN on a Maasai Boma classification task in the Tanzania region. The DeepSN takes a sequence of four seasonal data, each spanning three months, for Boma prediction. Through cross-season validation experiments, we compare various advanced DCNNs and select EfficientNet as the backbone for DeepSN, as it performs the best. DeepSN with an EfficientNet backbone achieves a significant 19% improvement in the F1 score compared to plain EfficientNet for the Boma classification task. This work introduces a versatile composite architecture capable of handling multi-temporal data efficiently, providing flexibility in choosing the most suitable feature extraction backbone. The performance of DeepSN demonstrates the viability of utilizing medium-resolution multi-temporal data instead of high-resolution images for diverse tasks.

Unified Transformer with Cross-Modal Mixture Experts for Remote-Sensing Visual Question Answering

Remote-sensing visual question answering (RSVQA) aims to provide accurate answers to remote sensing images and their associated questions by leveraging both visual and textual information during the inference process. However, most existing methods ignore the significance of the interaction between visual and language features, which typically adopt simple feature fusion strategies and fail to adequately model cross-modal attention, struggling to capture the complex semantic relationships between questions and images. In this study, we introduce a unified transformer with cross-modal mixture expert (TCMME) model to address the RSVQA problem. Specifically, we utilize the vision transformer (VIT) and BERT to extract visual and language features, respectively. Furthermore, we incorporate cross-modal mixture experts (CMMEs) to facilitate cross-modal representation learning. By leveraging the shared self-attention and cross-modal attention within CMMEs, as well as the modality experts, we effectively capture the intricate interactions between visual and language features and better focus on their complex semantic relationships. Finally, we conduct qualitative and quantitative experiments on two benchmark datasets: RSVQA-LR and RSVQA-HR. The results demonstrate that our proposed method surpasses the current state-of-the-art (SOTA) techniques. Additionally, we perform an extensive analysis to validate the effectiveness of different components in our framework.

Fruit ripeness identification using YOLOv8 model

Deep learning-based visual object detection is a fundamental aspect of computer vision. These models not only locate and classify multiple objects within an image, but they also identify bounding boxes. The focus of this paper's research work is to classify fruits as ripe or overripe using digital images. Our proposed model extracts visual features from fruit images and analyzes fruit peel characteristics to predict the fruit's class. We utilize our own datasets to train two "anchor-free" models: YOLOv8 and CenterNet, aiming to produce accurate predictions. The CenterNet network primarily incorporates ResNet-50 and employs the deconvolution module DeConv for feature map upsampling. The final three branches of convolutional neural networks are applied to predict the heatmap. The YOLOv8 model leverages CSP and C2f modules for lightweight processing. After analyzing and comparing the two models, we found that the C2f module of the YOLOv8 model significantly enhances classification results, achieving an impressive accuracy rate of 99.5%.

Neural encoding with unsupervised spiking convolutional neural network

Accurately predicting the brain responses to various stimuli poses a significant challenge in neuroscience. Despite recent breakthroughs in neural encoding using convolutional neural networks (CNNs) in fMRI studies, there remain critical gaps between the computational rules of traditional artificial neurons and real biological neurons. To address this issue, a spiking CNN (SCNN)-based framework is presented in this study to achieve neural encoding in a more biologically plausible manner. The framework utilizes unsupervised SCNN to extract visual features of image stimuli and employs a receptive field-based regression algorithm to predict fMRI responses from the SCNN features. Experimental results on handwritten characters, handwritten digits and natural images demonstrate that the proposed approach can achieve remarkably good encoding performance and can be utilized for “brain reading” tasks such as image reconstruction and identification. This work suggests that SNN can serve as a promising tool for neural encoding.